The Onset of Parisi’s Complexity in a Mismatched Inference Problem

Abstract

We show that a statistical mechanics model where both the Sherringhton–Kirkpatrick and Hopfield Hamiltonians appear, which is equivalent to a high-dimensional mismatched inference problem, is described by a replica symmetry-breaking Parisi solution.

1. Introduction

Beginning with Parisi’s seminal works on the Sherringhton–Kirkpatrick (SK) model [1,2], the ideas and the tools developed in spin glass theory spread across many other research fields such as computer science, probability theory, neural networks and others [3,4,5,6,7]. From a mathematical perspective, efforts to rigorously prove Parisi’s theory have yielded powerful techniques, such as interpolation methods [8,9], stochastic stability [10], and synchronization [11], which are currently instrumental in analyzing numerous disordered systems.

In this work, we will consider a family of mean-field spin glasses whose Hamiltonian contains two types of random interactions: the first is the SK type, while the second is a Hopfield model with a finite number of patterns. This class of models can also be interpreted as a high-dimensional inference problem known as a spiked Wigner model in a mismatched setting [12,13,14,15,16].

Our main result is a representation of the thermodynamic limit for the quenched pressure per particle in terms of a variational problem of Parisi type. The proof relies on two main ingredients: Guerra’s replica symmetry-breaking bound, which allows controlling the SK contribution, and adaptive interpolation, which is employed to linearize the Hopfield interaction. We start with a review of the SK model in Section 2 and then lay the ground for the inferential interpretation of the model under study in Section 3. In Section 4, we define and rigorously identify the exact solution of the model. Finally, we describe some interesting challenges for future investigations.

2. The SK Model

The SK model was introduced in the 1970s by D. Sherringhton and S. Kirkpatrick [17] and stands as an explicitly solvable mean-field spin glass. In their work, the authors discovered that the solution obtained through the replica symmetric (RS) approximation was not correct at low temperature. With a groundbreaking approach, Parisi identified a new type of solutions, nowadays called replica symmetry breaking (RSB), which proved to be correct at any temperature, thereby revealing a novel mathematical and physical structure [18].

The SK model is defined by its Hamiltonian, that is a function of N spins $\mathit{\sigma }={\left({\sigma }_i\right)}_{i\le N}\in {\{-1,1\}}^N$:

$$H_N^{SK}\left(\mathit{\sigma }\right)=-\frac{1}{\sqrt{2N}}\sum _{i,j\le N}{z}_{ij}{\sigma }_i{\sigma }_j$$

(1)

where $\mathbf{z}={\left(z_{ij}\right)}_{i,j\le N}$ is a collection of i.i.d. standard Gaussian. In physical terms, the couplings between pairs of spins can be ferromagnetic or antiferromagnetic with equal probability. Consider also a random variable $\xi$ with $\mathbb{E}\left|\xi \right|<\infty$ and a collection $\mathit{\xi }={\left({\xi }_i\right)}_{i\le N}\stackrel{\mathrm{iid}}{\sim }\xi$ representing random external fields acting on the spins. The Parisi formula is a representation for the large N limit of the pressure $p_N^{SK}$ defined by

$$p_N^{SK}(\beta ,h)=\frac{1}{N}log\sum _{\mathit{\sigma }\in {\{-1,1\}}^N}exp\left(-\beta H_N^{SK}\left(\mathit{\sigma }\right)+h\mathit{\sigma }·\mathit{\xi }\right)$$

(2)

In the definition (2), $(\beta ,h)\in {\mathbb{R}}_{>0}\times \mathbb{R}$ are fixed parameters, and the dependence on the realization of the random collections $\mathbf{z},\mathit{\xi }$ is kept implicit. One can prove [5] that $p_N^{SK}$ converges, for almost all realizations of the disorder, to its average ${\overline{p}}_N^{SK}(\beta ,h)=\mathbb{E}{p}_N^{SK}(\beta ,h)$. Notice that $\mathbb{E}$, taken after the logarithm, averages both the collections of $\mathbf{z}$ and $\mathit{\xi }$ that are called quenched variables. The Hamiltonian (1) can also be regarded as a centered Gaussian process with covariance

$$\mathbb{E}{H}_N^{SK}\left({\mathit{\sigma }}^1\right)H_N^{SK}\left({\mathit{\sigma }}^2\right)=\frac{N}{2}{q}_N^2\left({\mathit{\sigma }}^1,{\mathit{\sigma }}^2\right)$$

(3)

where

$$q_N({\mathit{\sigma }}^1,{\mathit{\sigma }}^2)=\frac{1}{N}\sum _{i=1}^N{\sigma }_i^1{\sigma }_i^2=\frac{1}{N}{\mathit{\sigma }}^1·{\mathit{\sigma }}^2.$$

(4)

$q_N({\mathit{\sigma }}^1,{\mathit{\sigma }}^2)$ is the overlap between two spin configurations ${\mathit{\sigma }}^1$ and ${\mathit{\sigma }}^2$.

The Parisi variational principle for the limiting pressure per particle of this model was proved after almost three decades of efforts, and it is mainly due to the works of Guerra [8] and Talagrand [19]. We hereby summarize these milestones in a single theorem.

Theorem 1

(Parisi Formula [8,19]). Let ${\mathcal{M}}_{[0,1]}$ be the space of probability measures on $[0,1]$, $\beta >0$ and $y\in \mathbb{R}$. Consider the Parisi functional, which is defined as

$$\begin{array}{c}\hfill \chi \in {\mathcal{M}}_{[0,1]}⟼\mathcal{P}(\chi ;\beta ,y)=log2+{\Phi }_{\chi }(0,y,\beta )-\frac{{\beta }^2}{2}{\int }_0^{1}dqq\chi \left([0,q]\right).\end{array}$$

(5)

where ${\Phi }_{\chi }(s,y,\beta )$ solves the PDE

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill & {\partial }_s{\Phi }_{\chi }=-\frac{{\beta }^2}{2}\left({\partial }_y^2{\Phi }_{\chi }+\chi \left([0,s]\right){\left({\partial }_y{\Phi }_{\chi }\right)}^2\right)\hfill \\ \hfill & {\Phi }_{\chi }(1,y,\beta )=logcoshy.\hfill \end{array}\right.\end{array}$$

(6)

The following holds

$$\underset{N\to \infty }{lim}{p}_N^{SK}(\beta ,h)=\underset{\chi \in {\mathcal{M}}_{[0,1]}}{inf}{\mathbb{E}}_{\xi }\mathcal{P}(\chi ;\beta ,h\xi )a.s.$$

(7)

The key tool for the proof is the (Gaussian) interpolation method, which is introduced in [9] in order to prove the existence of the large N limit of ${\overline{p}}_N^{SK}$.

The thermodynamic equilibrium induced by the pressure ${\overline{p}}_N^{SK}$ is called quenched equilibrium and is defined as follows. Physical quantities (e.g., energy) are functions of the disorder variables $\mathbf{z},\mathit{\xi }$ and the spin configurations $\mathit{\sigma }$. Given a function $f(\mathbf{z},\mathit{\xi },\mathit{\sigma })$, its equilibrium value is defined as

$$\mathbb{E}{\langle f\rangle }_N:=\mathbb{E}\sum _{\mathit{\sigma }\in {\{-1,1\}}^N}{G}_N\left(\mathit{\sigma }\right)f(\mathbf{z},\mathit{\xi },\mathit{\sigma })$$

(8)

where $G_N$ is the (random) Boltzmann–Gibbs distribution

$$\begin{array}{c}\hfill G_N\left(\mathit{\sigma }\right)={\displaystyle \frac{exp\left(-\beta H_N^{SK}\left(\mathit{\sigma }\right)+h\mathit{\sigma }·\mathit{\xi }\right)}{Z_N}}\end{array}$$

(9)

The measure $\mathbb{E}{\langle \rangle }_N$ is called a quenched measure and can be viewed as a two-step measuring process. Initially, for a given realization of the disorder variables $\mathbf{z},\mathit{\xi }$, one assumes that the system equilibrates according to the canonical Boltzmann–Gibbs distribution $G_N$ defining a (random) measure on the space of spin configurations. The expectation with respect to $G_N$ is denoted by ${\langle \rangle }_N$, namely

$${\langle f\rangle }_N:=\sum _{\mathit{\sigma }\in {\{-1,1\}}^N}{G}_N\left(\mathit{\sigma }\right)f(\mathbf{z},\mathit{\xi },\mathit{\sigma })$$

(10)

In probabilistic terms, $G_N$ defines a conditional measure given $\mathbf{z}$ and $\mathit{\xi }$. The remaining degrees of freedom $\mathbf{z}$, $\mathit{\xi }$ are then averaged according to their apriori distribution $\mathbb{E}$.

An important role is played by the concept of replicas. Replicas are i.i.d. samples from $G_N$ at fixed disorder. Hence, the equilibrium value of a function $f(\mathbf{z},\mathit{\xi },{\mathit{\sigma }}^1,\dots ,{\mathit{\sigma }}^n)$ of n replicas and the quenched variables $\mathbf{z},\mathit{\xi }$ is defined by

$$\mathbb{E}{\langle f\rangle }_N=\mathbb{E}\prod _{a\le n}\sum _{{\mathit{\sigma }}^a\in {\{-1,1\}}^N}{G}_N\left({\mathit{\sigma }}^a\right)f(\mathbf{z},\mathit{\xi },{\mathit{\sigma }}^1,\dots ,{\mathit{\sigma }}^n).$$

(11)

The computation of derivatives of ${\overline{p}}_N^{SK}$ shows, using integration by parts, that the SK model is fully characterized by the (joint) distribution of the overlap array ${\left(q_N({\mathit{\sigma }}^l,{\mathit{\sigma }}^{l^{\prime }})\right)}_{l,l^{\prime }\le n}\equiv {\left(q_{l,l^{\prime }}\right)}_{l,l^{\prime }\le n}$, namely the overlaps between any finite number n of replicas with respect to the measure (11). The main feature of the Parisi theory is the characterization of the mentioned joint measure by means of two structural properties:

(i)

It is uniquely determined by a one-dimensional marginal, namely the distribution of $q_{1,2}$;

(ii)

The distribution of three replicas has with a probability of one an ultrametric support

$$\underset{N\to \infty }{lim}\mathbb{E}〈\mathbb{1}\left(q_{1,2}\ge min\left(q_{1,3},q_{2,3}\right)\right)〉=1.$$

(12)

Despite having a mathematical proof of the Parisi Formula (7) for the SK model, (i) and (ii) have been rigorously proved only in the mixed p-spin model [6,20,21], an extension of the SK model, whose Hamiltonian contains also higher-order interactions (three-body, four-body, etc.).

One of the crucial instruments to achieve a rigorous control of the model is the so-called Ruelle Probability Cascades (RPCs), defined by Ruelle [22] when formalizing the properties of the Generalized Random Energy model of Derrida [23]. See also the characterization of RPC in terms of coalescent processes given in [24]. The first direct link between RPC and the SK model appeared in the work of Aizenman–Sims–Starr [25], where the authors found a representation of the thermodynamic limit of quenched pressure per particle in terms of the cavity fields distribution. This representation strongly suggested that if the thermodynamic limit of the overlap distribution is described by an RPC, then the Parisi formula is correct.

The first signal that the overlap array is described by an RPC was originally found by Aizenmann and Contucci in [10] with the identification of stochastic stability and by Ghirlanda and Guerra [26]. Both papers show an (infinite) set of identities for the moments of the overlap array distribution. It turns out that these identities actually imply that the support of the joint distribution of the overlap is ultrametric, as proved by Panchenko [27]. It should be noticed that Panchenko’s theorem requires identities for the overlap moments of all orders. The latter do not hold for the bare SK model, but it can be shown that there exists a perturbation of the Hamiltonian that forces the SK model to satisfy them without affecting the limit of the quenched pressure [28].

Once the validity of the Parisi Formula (7) is established, it is natural to ask for the properties of its solution. The uniqueness of the minimizer of (7) has been assessed by Auffinger and Chen [29], and its properties have been investigated for example in [30,31].

A relevant question about the minimizer is the following: for which values of the parameters $(\beta ,h)$ is the solution of (7) a Dirac-delta function ${\delta }_q$ for some $q\in [0,1]$? In this case, we say that the model is replica symmetric and the Parisi Formula (7) reads

$$\underset{N\to \infty }{lim}{p}_N^{SK}(\beta ,h)=\underset{q\in [0,1]}{inf}\left\{log2+{\mathbb{E}}_{z,\xi }logcosh\left(\beta z\sqrt{q}+h\xi \right)+\frac{{\beta }^2}{4}{(1-q)}^2\right\}.$$

(13)

The replica symmetric region can be identified [6,32] with the region of parameters $(\beta ,h)$ where the overlap is a self-averaging quantity, namely

$$\underset{N\to \infty }{lim}\mathbb{E}{〈{\left(q_{1,2}-q^{\ast }\right)}^2〉}_N=0$$

(14)

where $q^{\ast }$ is exactly the value that realizes the infimum in (13). The physics conjecture is that the replica symmetric region can be identified by the so called Almeida–Thouless [33]

$${\beta }^2{\mathbb{E}}_{z,\xi }{cosh}^{-4}\left(\beta z\sqrt{q^{\ast }}+h\xi \right)\le 1.$$

(15)

The above conjecture is proved only in the case of Gaussian external field ${\xi }_i\sim \mathcal{N}(0,1)$ [34]. An alternative characterization of the replica symmetric region has been obtained in [6,35]. If the minimizer corresponds to a non-trivial distribution (i.e., with non-zero variance), we say that replica symmetry breaking occurs, and the overlap is not a self-averaging quantity.

The Parisi formula has been extended to other mean field models with centered Gaussian interactions: vector spins [36], multispecies models [11,37,38], multiscale models [39,40]. Finally, we mention that the SK model fulfills a remarkable universality property: as long as $z_{ij}$’s are independent, centered, and with unit variance, the thermodynamic limit is still described by the Parisi solution [41].

In this work, we show that a class of non-centered Gaussian spin glasses admits an interpretation of high-dimensional inference that extends the celebrated correspondence between the spiked Wigner model and the SK model in the Nishimori line where replica symmetry is always fulfilled [3]. We show that the addition of an SK Hamiltonian to a Hopfield with a finite number of patterns can be mapped into a high-dimensional mismatched inference problem, where the statistician ignores the correct apriori distribution on the signal components they have to reconstruct. We shall see that even this slight mismatch may lead to the emergence of complexity, namely to the breakdown of replica symmetry, which is instead guaranteed under very mild hypotheses for optimal statisticians.

3. High-Dimensional Inference and Statistical Physics

High-dimensional inference aims at recovering a ground truth signal, $\mathit{\xi }$ in the following, that is usually a vector with a very large number of components from some noisy observations of it, which is denoted by $\mathbf{Y}$. The main feature of this setting is that the dimension of the signal, i.e., the number of real parameters to reconstruct, and the number of observations at disposal are a function of one another, typically a polynomial. For instance, for our purposes, $\mathit{\xi }$ will be a vector of ${\mathbb{R}}^N$ and $\mathbf{Y}$ will be an $N\times N$ matrix for a total of $N^2$ noisy observations. Hence, if the number of observations becomes large, the number of parameters to retrieve also does. Contrary to what happens in typical low-dimensional settings, where max-likelihood, or Maximum A Posteriori (MAP) approaches yield provably satisfactory reconstruction performances, in a high-dimensional setting, this is not always the case. In particular, one needs to devise another kind of more refined estimators that exploit the marginal posterior probabilities for each signal component.

Both approaches described above are Bayesian, and the knowledge of a prior distribution on the signal components can play a key role especially for high-dimensional problems. Furthermore, to compose the posterior measure for the entire signal, one needs the likelihood of the data, which is the probability of an outcome $\mathbf{y}$ of the variable $\mathbf{Y}$ given a certain ground truth realization $\mathit{\xi }=\mathbf{x}$. As we shall discuss soon, under certain hypotheses, the Bayesian approach highlights the correspondence of relevant information theoretic quantities with thermodynamic ones. Among the others, a key quantity is the mutual information between the signal $\mathit{\xi }$ and the observations $\mathbf{Y}$, which quantifies the residual amount of information left in $\mathbf{Y}$ about $\mathit{\xi }$ after the noise corruption. As intuition may suggest, the mutual information gives access to the best reconstruction error that is information theoretically achievable.

Finally, we stress that the high dimensionality of the problem can induce phase transition in some parameters of the model, like the so-called signal-to-noise ratio (SNR), that tunes the strength of the signal with respect to that of the noise in the observations.

3.1. Bayes-Optimality and Nishimori Identities

For the sake of simplicity, we start by considering a signal $\mathit{\xi }={\left({\xi }_i\right)}_{i\le N}\in {\mathbb{R}}^N$ of i.i.d. (independently and identically distributed) components ${\xi }_i\stackrel{\mathrm{iid}}{\sim }{P}_{\xi }$, where $P_{\xi }$ has a finite fourth moment. The observations at the disposal of a statistician can be modeled as a stochastic function of the ground-truth signal: $\mathbf{Y}=\mathbf{F}(\mathit{\xi };\mathbf{z})$, where $\mathbf{z}$ is the source of randomness or simply the noise. Knowing the function $\mathbf{F}$, from a Bayesian perspective, translates directly into having the likelihood of the model, namely the conditional distribution $d{P}_{\mathbf{Y}|\mathit{\xi }=\mathbf{x}}\left(\mathbf{y}\right)=p_{\mathbf{Y}|\mathit{\xi }=\mathbf{x}}\left(\mathbf{y}\right)d\mathbf{y}$, which we assume to have a density $p_{\mathbf{Y}|\mathit{\xi }=\mathbf{x}}\left(\mathbf{y}\right)$ over the Lebesgue measure. Observe that the likelihood is strongly affected by the nature of the noise.

According to Bayes’ rule, the posterior distribution of $\mathit{\xi }$ given the data is:

$$\begin{array}{c}\hfill d{P}_{\mathit{\xi }\mid \mathbf{Y}=\mathbf{y}}\left(\mathbf{x}\right)=\frac{p_{\mathbf{Y}|\mathit{\xi }=\mathbf{x}}\left(\mathbf{y}\right)d{P}_{\xi }\left(\mathbf{x}\right)}{\mathcal{Z}\left(\mathbf{y}\right)},\mathcal{Z}\left(\mathbf{y}\right)=\int p_{\mathbf{Y}|\mathit{\xi }=\mathbf{x}}\left(\mathbf{y}\right)d{P}_{\xi }\left(\mathbf{x}\right),\end{array}$$

(16)

where $d{P}_{\xi }\left(\mathbf{x}\right)={\prod }_{i\le N}d{P}_{\xi }\left(x_i\right)$, and $\mathcal{Z}\left(\mathbf{y}\right)$ is the probability of a given realization of the data, which is sometimes also called evidence. In practice, the above posterior, which would be ideal to perform inference, is rarely available, and the statistician is not aware either of the likelihood or of the correct prior distribution for the signal, or even both. This motivates the following definition of a special inference setting:

Definition 1

(Bayes optimality). The statistician is said to be Bayes optimal, or in the Bayes-optimal setting, if they are aware both of $P_{\xi }$ and $\mathbf{F}(·;\mathbf{z})$; namely, they have access to the posterior (16).

The above is saying that an optimal statistician knows everything about the model except for the ground truth $\mathit{\xi }$ itself. The Bayes-optimal setting is thus often used as a theoretical framework to establish the information theoretical limits. Indeed, it is known that the mean square error between the ground truth and an estimator $\widehat{\mathit{\xi }}\left(\mathbf{y}\right)$

$$\begin{array}{c}\hfill \mathrm{MSE}\left(\widehat{\mathit{\xi }}\right)=\mathbb{E}{\parallel \mathit{\xi }-\widehat{\mathit{\xi }}\left(\mathbf{y}\right)\parallel }^2\end{array}$$

(17)

is minimized by an optimal statistician that can use the posterior mean as an estimator, yielding the minimum mean square error (MMSE)

$$\begin{array}{c}\hfill \mathrm{MMSE}=\mathbb{E}\parallel \mathit{\xi }-{\mathbb{E}}_{\mathit{\xi }\mid \mathbf{Y}}{\mathit{\xi }\parallel }^2.\end{array}$$

(18)

In the following, we shall denote averages with respect to the posterior as ${\langle ·\rangle }_{\mathbf{Y}}$.

Another important consequence of this setting is the so-called Nishimori identities, which can be stated as follows. Given any continuous bounded function f of the data $\mathbf{Y}$, the ground truth $\mathit{\xi }$ and $n-1$ i.i.d. samples from the posterior ${\left({\mathbf{x}}^{\left(k\right)}\right)}_{k=2}^n$, one has

$$\begin{array}{c}\hfill \mathbb{E}{\langle f(\mathbf{Y},\mathit{\xi },{\left({\mathbf{x}}^{\left(k\right)}\right)}_{k=2}^n)\rangle }_{\mathbf{Y}}=\mathbb{E}{\langle f(\mathbf{Y},{\mathbf{x}}^{\left(1\right)},{\left({\mathbf{x}}^{\left(k\right)}\right)}_{k=2}^n)\rangle }_{\mathbf{Y}},,\end{array}$$

(19)

where ${\mathbf{x}}^{\left(1\right)}\sim P_{\mathit{\xi }\mid \mathbf{Y}}$. An elementary proof can be found in [42]. These identities are enforcing a symmetry between replicas drawn from the posterior and the ground truth. For instance, a direct application of the Nishimori identities yields

$$\begin{array}{c}\hfill {\mathrm{MMSE}=\mathbb{E}\parallel \mathit{\xi }-\langle \mathbf{x}\rangle }_{\mathbf{Y}}{\parallel }^2=\mathbb{E}\langle \parallel \mathbf{x}-{\langle \mathbf{x}\rangle }_{\mathbf{Y}}{{\parallel }^2\rangle }_{\mathbf{Y}}.\end{array}$$

(20)

It is important to stress that, as it can be seen from the above equation, an optimal statistician is actually able to compute the minimum mean square error using their posterior.

At this point, the reader will have noticed a similarity with the Statistical Mechanics formalism. In fact, it is possible to interpret $\mathcal{Z}\left(\mathbf{y}\right)$ as the partition function of a model with Hamiltonian $-log{p}_{\mathbf{Y}\mid \mathit{\xi }=\mathbf{x}}\left(\mathbf{y}\right)$ and unit inverse absolute temperature. The pressure per particle of such a model would thus be

$$\begin{array}{c}\hfill p_N\left(\mathbf{y}\right)=\frac{1}{N}log\mathcal{Z}\left(\mathbf{y}\right),{\overline{p}}_N={\mathbb{E}}_{\mathbf{Y}}{p}_N\left(\mathbf{Y}\right)=-\frac{1}{N}\mathcal{H}\left(\mathbf{Y}\right),\end{array}$$

(21)

namely minus the Shannon entropy of the data per signal component, which is related to the mutual information

$$\begin{array}{c}\hfill \frac{1}{N}{I}_N(\mathbf{Y};\mathit{\xi })=\frac{1}{N}\mathcal{H}\left(\mathbf{Y}\right)-\frac{1}{N}\mathcal{H}(\mathbf{Y}\mid \mathit{\xi }).\end{array}$$

(22)

The contribution coming from the conditional entropy $\mathcal{H}(\mathbf{Y}\mid \mathit{\xi })$ can be regarded as due only to the noise, since for fixed $\mathit{\xi }$, the only randomness in $\mathbf{Y}$ is due to $\mathbf{Z}$.

We stress here that Bayes optimality, and the Nishimori identities, under rather mild hypotheses [43] are enough to grant replica symmetry in the model, i.e., concentration of the order parameters in the model. For the models we are interested in, the latter can be shown to imply finite-dimensional variational principles for the limiting mutual information.

3.2. The Spiked Wigner Model

The spiked Wigner model (WSM) was first introduced in [44] as a model for Principal Component Analysis (PCA), and since then, it was widely studied in recent literature. Without pretension of being exhaustive, we refer the interested reader to [42,45,46,47,48,49,50,51]. For our purposes, we restrict ourselves to the case where the signal is an N-dimensional vector of $\pm 1$ s, drawn from a Rademacher distribution ${\xi }_i\stackrel{\mathrm{iid}}{\sim }{P}_{\xi }=({\delta }_{-1}+{\delta }_1)/2$. The function $\mathbf{F}(;\mathbf{z})$ is a Gaussian channel, namely

$$\begin{array}{c}\hfill Y_{ij}=\sqrt{\frac{\mu }{2N}}{\xi }_i{\xi }_j+z_{ij}\end{array}$$

(23)

where $z_{ij}\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,1)$, and $\mu$ is a positive parameter called the signal-to-noise ratio. The statistician is tasked with the recovery of $\mathit{\xi }$ given the observations $\mathbf{Y}$. The Bayes-optimal posterior measure for this inference problem can be written directly as a Boltzmann–Gibbs random measure thanks to the Gaussian nature of the likelihood:

$$\begin{array}{c}\hfill G_N\left(\mathit{\sigma }\right)=\frac{1}{\mathcal{Z}(\mathbf{z},\mathit{\xi })}{e}^{-H_N(\mathit{\sigma };\mathbf{z},\mathit{\xi })},-H_N(\mathit{\sigma };\mathbf{z},\mathit{\xi })=\sum _{i,j=1}^N\sqrt{\frac{\mu }{2N}}{z}_{ij}{\sigma }_i{\sigma }_j+\frac{\mu }{2N}{\sigma }_i{\xi }_i{\sigma }_j{\xi }_j\end{array}$$

(24)

where we have already exploited the fact that ${\left({\xi }_i\right)}^2={\sigma }_i^2=1$. We are denoting the posterior samples with $\mathit{\sigma }$. Since the quantity we are interested in is the quenched pressure of this model

$$\begin{array}{c}\hfill {\overline{p}}_N=\frac{1}{N}\mathbb{E}log\sum _{\mathit{\sigma }\in {\{-1,1\}}^N}exp\left[-H_N(\mathit{\sigma };\mathbf{z},\mathit{\xi })\right]\end{array}$$

(25)

that is connected to the mutual information $I_N(\mathbf{Y};\mathit{\xi })/N$ by a simple shift with an additive constant, we are allowed to perform a gauge transformation without altering its value:

$$\begin{array}{c}\hfill z_{ij}\mapsto z_{ij}{\xi }_i{\xi }_j,{\sigma }_i{\sigma }_j\mapsto {\sigma }_i{\sigma }_j{\xi }_i{\xi }_j.\end{array}$$

(26)

This results in a Hamiltonian that is now independent of the original ground-truth signal

$$\begin{array}{c}\hfill -H_N^{\prime }(\mathit{\sigma };\mathbf{z})=\sum _{i,j=1}^N\left(\sqrt{\frac{\mu }{2N}}{z}_{ij}+\frac{\mu }{2N}\right){\sigma }_i{\sigma }_j\end{array}$$

(27)

and the coupling between spins are Gaussian random variables with a mean equal to their variance. This condition identifies a peculiar region of the phase space of a spin-glass model, which is called Nishimori line. In fact, the Nishimori identities were first discovered and studied in the context of gauge spin-glasses. Despite looking simpler, the above model retains most of the features we need for our study.

For inference models with additive Gaussian noise, like the one above, it is possible to prove the so-called I-MMSE relation:

$$\begin{array}{c}\hfill \frac{d}{d\mu }\frac{I_N(\mathbf{Y};\mathit{\xi })}{N}=\frac{1}{4N^2}\mathbb{E}{\parallel \mathit{\xi }{\mathit{\xi }}^{⊺}-\langle \mathit{\sigma }{\mathit{\sigma }}^{⊺}\rangle \parallel }_F^2,\end{array}$$

(28)

where ${\parallel ·\parallel }_F$ is the Frobenius norm and $\langle ·\rangle$ denotes the expectation with respect to the Boltzmann–Gibbs measure induced by (27). Hence, once the mutual information is known, the MMSE can be accessed through a derivative with respect to the signal-to-noise ratio. A clarification is in order here: the above is the MMSE on the reconstruction of the rank-one matrix $\mathit{\xi }{\mathit{\xi }}^{⊺}$, because, due to flip symmetry, here we do not have any actual information on the single vector $\mathit{\xi }$, but only on the spike $\mathit{\xi }{\mathit{\xi }}^{⊺}$.

3.3. Sub-Optimality and Replica Symmetry Breaking

There are several ways to break Bayes optimality. Some examples are that the statistician does not know the signal-to-noise ratio $\mu$ [13,52]; the statistician adopts a likelihood different from that of the true model [14]; the statistician adopts a wrong prior [12,53]; combinations of the previous and many others. We will focus on the mismatching priors case, where the statistician not only adopts a wrong prior on the ground-truth elements, but they are not aware of the rank of the spiked matrix hidden inside the noise, which is denoted by M. The rest is assumed to be known. The channel of the inference problem is

$$\begin{array}{c}\hfill Y_{ij}=\sqrt{\frac{\mu }{2N}}\sum _{k=1}^M{\xi }_i^{\left(k\right)}{\xi }_j^{\left(k\right)}+z_{ij}.\end{array}$$

(29)

If the statistician assumes a Rademacher prior to the signal components and a rank-one hidden matrix, they will write a posterior in the form

$$\begin{array}{c}\hfill {/P}_{\mathit{\xi }|\mathbf{z},\mathit{\xi }}\left(\mathit{\sigma }\right)=\frac{1}{/\mathcal{Z}(\mathbf{z},\mathit{\xi })}exp\left(-H_N(\mathit{\sigma };\mathbf{z},\mathit{\xi })\right)\end{array}$$

(30)

where

$$\begin{array}{c}\hfill -H_N(\mathit{\sigma };\mathbf{z},\mathit{\xi })=\sum _{i,j=1}^N\left(\sqrt{\frac{\mu }{2N}}{z}_{ij}+\frac{\mu }{2N}\sum _{k=1}^P{\xi }_i^{\left(k\right)}{\xi }_j^{\left(k\right)}\right){\sigma }_i{\sigma }_j\end{array}$$

(31)

The slash on quantities emphasizes that they are not the Bayes-optimal ones. In this setting, one can no longer rely on the Nishimori identities, and in principle, replica symmetry is no longer guaranteed. On the contrary, as we shall argue later on, a mismatch in the prior only is already sufficient to cause replica symmetry breaking.

4. The Model

Let M be a fixed integer and $k\in \{1,\dots ,M\}$. Consider two independent random collections ${\left(z_{ij}\right)}_{i,j\le N}\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,1)$ and ${\left({\xi }_i^{\left(k\right)}\right)}_{i\le N}^{k\le M}\stackrel{\mathrm{iid}}{\sim }{P}_{\xi }$ where $P_{\xi }$ is such that $\mathbb{E}\left[{\xi }^4\right]<\infty$. The above random collections play the role of quenched disorder in the model. Consider N Ising spins $\mathit{\sigma }={\left({\sigma }_i\right)}_{i\le N}\in {\{+1,-1\}}^N$ and the Hamiltonian function

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill H_N(\mathit{\sigma };\mu ,\nu ,\lambda )& \equiv H_N\left(\mathit{\sigma }\right)=H_N^{int}\left(\mathit{\sigma }\right)-\mathit{h}·\mathit{\sigma }\hfill \\ \hfill & =-\sum _{i,j=1}^N\left(\sqrt{\frac{\mu }{2N}}{z}_{ij}+\frac{\nu }{2N}\sum _{k\le M}{\xi }_i^{\left(k\right)}{\xi }_j^{\left(k\right)}\right){\sigma }_i{\sigma }_j-\sum _{k\le M}{\lambda }_k\sum _{i=1}^N{\xi }_i^{\left(k\right)}{\sigma }_i\hfill \end{array}\end{array}$$

(32)

with $\mu ,\nu \ge 0,\lambda ={\left({\lambda }_k\right)}_{k\le M}\in {\mathbb{R}}^M$. Here, $H_N^{int}$ is the interacting part while

$$\mathit{h}={\left(h_i\right)}_{i\le N}\equiv \mathit{h}(\lambda ,\mathit{\xi })={\left(\sum _{k\le M}{\lambda }_k{\xi }_i^{\left(k\right)}\right)}_{i\le N}$$

(33)

denotes the random external field acting on the spins. The Hamiltonian (32) is determined by the choice of $M,\mu ,\nu ,\lambda$ and $P_{\xi }$. For $\mu =\nu$, the interaction term $H_N^{int}$ coincides with the Hamiltonian (31). Note that for some special choices of the parameters, we recover some well-known spin glass models:

• $\nu =0$ gives the SK model (1) at $\beta =\sqrt{\mu }$ and random external field $\mathit{h}$.
• $\mu =0$ gives the Hopfield model [6,7,18] with a finite number of patterns ${\left({\mathit{\xi }}^{\left(k\right)}\right)}^{k\le M}$.
• $M=1,\nu =\mu ,{\lambda }_1=0$ and $P_{\xi }=\frac{1}{2}({\delta }_{-1}+{\delta }_{-1})$ gives the SK model on the Nishimori line (27). As we have seen in Section 3, the latter can be also viewed as a spiked Wigner model in the Bayesian-optimal setting.

Notice that the entire model can be interpreted as a Hopfield model where the traditional Hebbian matrix ${\sum }_{k\le M}{\xi }_i^{\left(k\right)}{\xi }_j^{\left(k\right)}$ is corrupted by Gaussian noise. Furthermore, if the Hebbian coupling is replaced by a constant matrix, the model reduces to an SK model with the addition of a ferromagnetic interaction, and it was studied in [54].

Our main result is the computation of the thermodynamic limit of the pressure per particle

$$\begin{array}{c}\hfill p_N(\mu ,\nu ,\lambda )=\frac{1}{N}log\sum _{\mathit{\sigma }\in {\{-1,1\}}^N}{e}^{-H_N\left(\mathit{\sigma }\right)}\end{array}$$

(34)

whose variance can be shown to converge to 0 as an $O\left(N^{-1}\right)$, namely:

Lemma 1.

Assume $\mathbb{E}{\xi }_1^4<\infty$. Then, for any $\mu ,\nu \in \mathbb{R}$ and $\lambda \in {\mathbb{R}}^K$

$$\begin{array}{c}\hfill \mathbb{E}{\left[p_N(\mu ,\nu ,\lambda )-\mathbb{E}{p}_N(\mu ,\nu ,\lambda )\right]}^2\le \frac{K}{N}\end{array}$$

(35)

where K is a suitable positive constant.

We thus focus on ${\overline{p}}_N(\mu ,\nu ,\lambda )=\mathbb{E}{p}_N(\mu ,\nu ,\lambda )$. The proof of this lemma makes use of the Efron–Stein concentration inequality to bound the variance, and it is simple but tedious. It follows closely that of ([12], Lemma 9). We are now in a position to state our main theorem:

Theorem 2

(Variational solution). If $\mathbb{E}\left[{\xi }^4\right]<+\infty$ then

$$\begin{array}{c}\hfill \underset{N\to \infty }{lim}{\overline{p}}_N(\mu ,\nu ,\lambda )=\underset{x\in {\mathbb{R}}^M}{sup}\phi (x;\mu ,\nu ,\lambda )\end{array}$$

(36)

where

$$\begin{array}{c}\hfill \phi (x;\mu ,\nu ,\lambda ):=-\frac{{\nu \left|x\right|}^2}{2}+\mathbb{E}\mathcal{P}\left(\sqrt{\mu },h(x,\lambda ,\xi )\right),x={\left(x_k\right)}_{k\le M}\in {\mathbb{R}}^M\end{array}$$

(37)

and $\mathcal{P}$ is the Parisi functional (5) with a random external field

$$\begin{array}{c}\hfill h(x,\lambda ,\xi )=\sum _{k\le M}({\lambda }_k+\nu x_k){\xi }^{\left(k\right)}\end{array}$$

(38)

and $\mathbb{E}$ denotes the expectation with respect to ${\xi }^{\left(k\right)}\stackrel{\mathrm{iid}}{\sim }{P}_{\xi }$. The consistency equations are

$$x_k={\displaystyle \frac{\partial }{\partial x_k}}\mathbb{E}\mathcal{P}\left(\sqrt{\mu },h(x,\lambda ,\xi )\right),k\le M$$

(39)

Moreover, there exists $C>0$ such that for any $k\le M$, one has $|x_k|\le C$ and the supremum in (36) can be restricted to ${[-C,C]}^M$.

The proof of the theorem is based on the concentration of the Mattis magnetization, which is the normalized scalar product between a spin-configuration (or sample from the wrong posterior measure) and one of the ${\mathit{\xi }}^{\left(k\right)}$:

$$m_N\left(\mathit{\sigma }\right|{\xi }^{\left(k\right)})=\frac{1}{N}\sum _{i=1}^N{\sigma }_i{\xi }_i^{\left(k\right)}=\frac{1}{N}\mathit{\sigma }·{\mathit{\xi }}^{\left(k\right)}.$$

(40)

The Hamiltonian can thus be rewritten using (40) in the following form:

$$\begin{array}{c}\hfill H_N\left(\mathit{\sigma }\right)=\sqrt{\mu }{H}_N^{SK}\left(\mathit{\sigma }\right)-N\sum _{k\le M}\left[\frac{\nu }{2}{m}_N^2\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})+{\lambda }_k{m}_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\right]\end{array}$$

(41)

The Mattis magnetization, in fact, plays the role of an order parameter for this model. The concentration we can prove is only an integral average over some suitably small magnetic fields, which is still sufficient for our purposes:

Proposition 1

(Concentration of Mattis Magnetizations). Consider a k such that $k\le M$. Let ${ϵ}_k\in [s_N,2s_N]$ with $s_N=\frac{1}{2}{N}^{-\alpha }$, $\alpha \in (0,1/(2M\left)\right)$ for all $k\le M$. For any $y\in \mathbb{R}$, we denote by ${\langle ·\rangle }_{N,y}$ the Boltzmann–Gibbs measure induced by the Hamiltonian $H_{N,y}\left(\mathit{\sigma }\right)=H_N\left(\mathit{\sigma }\right)-y\mathit{\sigma }·{\mathit{\xi }}^{\left(k\right)}$. Then

$$\begin{array}{c}\hfill \underset{N\to \infty }{lim}\frac{1}{s_N^M}{\int }_{s_N}^{2s_N}\prod _{\ell \le M}d{ϵ}_{\ell }\mathbb{E}{〈{\left(m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})-\mathbb{E}{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,ϵ}\right)}^2〉}_{N,ϵ}=0,\end{array}$$

(42)

for all $\mu ,\nu \ge 0$ and $\lambda \in {\mathbb{R}}^M$.

We shall omit the proof of the above result as it is completely analogous to the one in [12]. We will need an intermediate lemma that leads to it (see Lemma 2 later) together with a second key ingredient: the adaptive interpolation technique [48] combined with Guerra’s replica symmetry-breaking upper bound for the quenched pressure of the SK model [8].

Proof of Theorem 2.

Here, we outline the main steps of the proof of the variational principle for the thermodynamic limit. The proof is achieved via two bounds that match in the $N\to \infty$ limit. Let us start by defining the interpolating Hamiltonian

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill H_N(t;\mathit{\sigma }):& =H_N(\mathit{\sigma };\mu ,(1-t)\nu ,\lambda +R_{ϵ}\left(t\right))=\hfill \\ \hfill & =\sqrt{\mu }{H}_N^{SK}\left(\mathit{\sigma }\right)-(1-t)\frac{N\nu }{2}\sum _{k\le M}{m}_N^2\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})-N\sum _{k\le M}({\lambda }_k+R_{ϵ}^{\left(k\right)}\left(t\right))m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\hfill \end{array}\end{array}$$

(43)

where $R_{ϵ}={\left(R_{ϵ}^{\left(k\right)}\right)}_{k\le M}$ and

$$\begin{array}{c}\hfill R_{ϵ}^{\left(k\right)}\left(t\right)={ϵ}_k+\nu {\int }_0^{t}ds{r}_{ϵ}^{\left(k\right)}\left(s\right),{ϵ}_k\in [s_N,2s_N],s_N=\frac{N^{-\alpha }}{2}\end{array}$$

(44)

with $\alpha \in (0,1/(2M\left)\right)$ and where the interpolating functions $r_{ϵ}^{\left(k\right)}$, that must be continuously differentiable in $[0,1]$ and non negative, will be suitably chosen. With this interpolation, one is able to prove the following sum rule:

Proposition 2.

The following sum rule holds:

$$\begin{array}{c}\hfill {\overline{p}}_N(\mu ,\nu ,\lambda )={\overline{p}}_N^{SK}(\sqrt{\mu },\lambda +R_{ϵ}\left(1\right))-\frac{\nu }{2}{\int }_0^{1}dt\sum _{k\le M}[r_{ϵ}^{\left(k\right)2}\left(t\right)-{\Delta }_{ϵ}^{\left(k\right)}\left(t\right)]+O\left(s_N\right)\end{array}$$

(45)

where

$$\begin{array}{c}\hfill {\Delta }_{ϵ}^{\left(k\right)}\left(t\right):=\mathbb{E}{〈{\left(m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})-r_{ϵ}^{\left(k\right)}\left(t\right)\right)}^2〉}_{N,R_{ϵ}\left(t\right)}.\end{array}$$

(46)

The proof consists of the computation of the derivative of the interpolating pressure related to the model (43). It follows closely that of ([12], Proposition 7), to which we refer the interested reader. Since the remainder ${\Delta }_{ϵ}^{\left(k\right)}$ is non-negative, the above proposition already yields a bound for the quenched pressure of our model when we choose $r_{ϵ}^{\left(k\right)}=x_k\in \mathbb{R}$ constant:

$$\begin{array}{c}\hfill \underset{N\to \infty }{lim\; inf}{\overline{p}}_N(\mu ,\nu ,\lambda )\ge \underset{x\in {\mathbb{R}}^M}{sup}\phi (x;\mu ,\nu ,\lambda )\end{array}$$

(47)

where we used Lipschitz continuity of the SK pressure in the magnetic fields.

The upper bound requires more attention. First, we notice that ${\overline{p}}_N^{SK}(\sqrt{\mu },\lambda +R_{ϵ}\left(1\right))$ is convex in the magnetic fields and that $R_{ϵ}^{\left(k\right)}\left(1\right)={\int }_0^{1}dt({ϵ}_k+\nu r_{ϵ}^{\left(k\right)}\left(t\right))$. Hence, we can use Jensen’s inequality and Lipschitz continuity of ${\overline{p}}^{SK}$ to obtain:

$$\begin{array}{c}\hfill {\overline{p}}_N(\mu ,\nu ,\lambda )\le {\int }_0^{1}dt\left[{\overline{p}}_N^{SK}(\sqrt{\mu },\lambda +r_{ϵ}\left(t\right))-\nu \sum _{k_{\le }M}\frac{r_{ϵ}^{\left(k\right)2}\left(t\right)}{2}\right]+\frac{\nu }{2}\sum _{k\le M}{\int }_0^1{\Delta }_{ϵ}^{\left(k\right)}\left(t\right)dt+O\left(s_N\right).\end{array}$$

(48)

Now, we use Guerra’s bound for the SK pressure, that, importantly, is uniform in N, and we average over $\mathbf{ϵ}$ on both sides

$$\begin{array}{c}{\overline{p}}_N(\mu ,\nu ,\lambda )\le {\mathbb{E}}_{\mathbf{ϵ}}{\int }_0^1\phi (r_{ϵ}\left(t\right);\mu ,\nu ,\lambda )dt+\frac{\nu }{2}\sum _{k\le M}{\mathbb{E}}_{\mathbf{ϵ}}{\int }_0^1{\Delta }_{ϵ}^{\left(k\right)}\left(t\right)dt+O\left(s_N\right)\le \hfill \\ \hfill \le \underset{x\in {\mathbb{R}}^M}{sup}\phi (x;\mu ,\nu ,\lambda )+\frac{\nu }{2}\sum _{k\le M}{\mathbb{E}}_{\mathbf{ϵ}}{\int }_0^1{\Delta }_{ϵ}^{\left(k\right)}\left(t\right)dt+O\left(s_N\right).\end{array}$$

(49)

What remains to do is to prove that ${\mathbb{E}}_{\mathbf{ϵ}}{\Delta }_{ϵ}^{\left(k\right)}\left(t\right)\stackrel{N\to \infty }{\to }0$ for a proper choice of the interpolating functions $R_{ϵ}$. The choice is made through a system of coupled ODEs

$$\begin{array}{c}\hfill {\dot{R}}_{ϵ}^{\left(k\right)}\left(t\right)\equiv \nu r_{ϵ}^{\left(k\right)}\left(t\right)=\nu \mathbb{E}{\langle m_N(\mathit{\sigma }\mid {\mathit{\xi }}^{\left(k\right)})\rangle }_{N,R_{ϵ}\left(t\right)},R_{ϵ}^{\left(k\right)}\left(0\right)={ϵ}_k,\forall k\le M.\end{array}$$

(50)

One can easily check that the above system is regular enough to admit a unique solution on the interval $t\in [0,1]$. In this case, the remainder to push to 0 would appear as

$$\begin{array}{c}\hfill \frac{1}{s_N^M}{\int }_{s_N}^{2s_N}\prod _{\ell \le M}d{ϵ}_{\ell }\mathbb{E}{〈{\left(m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})-\mathbb{E}{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,R_{ϵ}\left(t\right)}\right)}^2〉}_{N,R_{ϵ}\left(t\right)}.\end{array}$$

(51)

The goal is now to apply a concentration lemma here:

Lemma 2.

Let $y\in [y_1,y_2],\delta \in (0,1)$ and denote by ${\langle ·\rangle }_{N,y}$ the Boltzmann–Gibbs expectation associated to the Hamiltonian $H_N(\sigma ;\mu ,\nu ,\lambda +y{\mathbf{e}}_k)$ where $k\le M$ and ${\mathbf{e}}_k$ is the k-th canonical basis vector of ${\mathbb{R}}^M$. Then

$$\begin{array}{cc}\hfill & \mathbb{E}{〈{\left(m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})-{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,y}\right)}^2〉}_{N,y}=\frac{1}{N}\frac{d^2}{d{y}^2}{\overline{p}}_N(\mu ,\nu ,\lambda +y{\mathbf{e}}_k)\hfill \end{array}$$

(52)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \mathbb{E}\left[{\left({\langle m_N\left(\mathit{\sigma }\right|\mathit{\xi })\rangle }_{N,y}-\mathbb{E}{\langle m_N\left(\mathit{\sigma }\right|\mathit{\xi })\rangle }_{N,y}\right)}^2\right]\le \frac{12K}{{\delta }^{2}N}+\hfill \\ & +8\sqrt{a}\frac{d}{dy}\left[{\overline{p}}_N(\mu ,\nu ,\lambda +(y+\delta ){\mathbf{e}}_k)-{\overline{p}}_N(\mu ,\nu ,\lambda +(y-\delta ){\mathbf{e}}_k)\right]\hfill \end{array}\end{array}$$

(53)

with K a positive constant.

Notice that the integral in (51) is over $ϵ$ and not over the effective magnetic field of the model, which is instead $R_{ϵ}\left(t\right)$. Nevertheless, we can integrate over the magnetic fields $R_{ϵ}\left(t\right)$ with a change of variables. This involves a Jacobian that is larger than 1. In fact, thanks to Liouville’s theorem ([55], Corollary 3.1, Chapter V), one can prove that

$$\begin{array}{c}det\frac{\partial R_{ϵ}\left(t\right)}{\partial ϵ}=exp\left[{\int }_0^t\nu \sum _{k\le M}\frac{\partial }{\partial R_{ϵ}^{\left(k\right)}\left(s\right)}\mathbb{E}{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,R_{ϵ}\left(s\right)}ds\right]=\hfill \\ \hfill exp\left[{\int }_0^{t}N\nu \sum _{k\le M}\mathbb{E}{\langle {\left(m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})-\mathbb{E}{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,R_{ϵ}\left(s\right)}\right)}^2\rangle }_{N,R_{ϵ}\left(s\right)}ds\right]\ge 1,\end{array}$$

(54)

when $\nu \ge 0$.

This allows us to bound the thermal fluctuations in (51) using (52) and then Liouville’s theorem:

$$\begin{array}{c}{\Delta }_T^{\left(k\right)}:=\frac{1}{s_N^M}{\int }_{s_N}^{2s_N}\prod _{\ell \le M}d{ϵ}_{\ell }\mathbb{E}{〈{\left(m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})-{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,R_{ϵ}\left(t\right)}\right)}^2〉}_{N,R_{ϵ}\left(t\right)}\le \hfill \\ \le \frac{1}{s_N^M}\prod _{\ell \le M}{\int }_{R_{s_N}^{(\ell )}\left(t\right)}^{R_{2s_N}^{(\ell )}\left(t\right)}d{h}_{\ell }\frac{1}{N}\frac{d^2}{d{h}_k^2}{\overline{p}}_N(\mu ,\nu ,\lambda +\mathbf{h})=\\ \hfill =\frac{1}{N{s}_N^M}\prod _{\ell \le M,\ne k}{\int }_{R_{s_N}^{(\ell )}\left(t\right)}^{R_{2s_N}^{(\ell )}\left(t\right)}d{h}_{\ell }\left[\mathbb{E}{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,\mathbf{h};h_k=R_{2s_N}^{\left(k\right)}}-\mathbb{E}{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,\mathbf{h};h_k=R_{s_N}^{\left(k\right)}}\right]\end{array}$$

(55)

Since ${\xi }_i$ has a bounded second moment, using Cauchy–Schwarz inequality, one can show that $|\mathbb{E}{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,\mathbf{h}}|$ is uniformly bounded by a constant C. Hence, $|R_{ϵ}^{\left(k\right)}\left(t\right)|\le {ϵ}_k+Ct\nu \le 1+C\nu$ for any $k\le M$ by construction (recall (44) and (50)). Therefore, ${\Delta }_T=O\left(\frac{1}{N{s}_N^M}\right)$.

The fluctuations induced by the disorder can be bounded in a very similar fashion using (53):

$$\begin{array}{c}\hfill {\Delta }_D^{\left(k\right)}:=\frac{1}{s_N^M}{\int }_{s_N}^{2s_N}\prod _{\ell \le M}d{ϵ}_{\ell }\mathbb{E}{\left({\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,R_{ϵ}\left(t\right)}-\mathbb{E}{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^{\left(k\right)})\rangle }_{N,R_{ϵ}\left(t\right)}\right)}^2=O\left(\frac{1}{N{\delta }^2}+\frac{\delta }{s_N^M}\right)\end{array}$$

(56)

Hence, overall (51), that equals ${\Delta }_T^{\left(k\right)}+{\Delta }_D^{\left(k\right)}$ is a $O\left(\frac{1}{N{s}_N^M}+\frac{1}{N{\delta }^2}+\frac{\delta }{s_N^M}\right)$. $\delta$ can be chosen as a function of N in order to optimize the convergence rate: $\delta =s_N^{2M/3}{N}^{-1/3}$. Using Fubini’s theorem in (49) to exchange the t and $\mathbf{ϵ}$ averages and then Dominated Convergence, one concludes the proof. □

From the the variational problem (36), we can deduce also the differentiability properties of the limiting pressure obtaining the average values of the relevant thermodynamic quantities of the model:

Corollary 1.

Let $p(\mu ,\nu ,\lambda )={lim}_{N\to \infty }{\overline{p}}_N(\mu ,\nu ,\lambda )$, and ${\Omega }_{\mu ,\nu ,\lambda }={\mathrm{argmax}}_{x\in {[-C,C]}^M}\phi (x;\mu ,\nu ,\lambda )$. Then

• If ${\Omega }_{\mu ,\nu ,\lambda }=\left\{\overline{x}\right\}$ is a singleton with $\overline{x}={\left({\overline{x}}_k\right)}_{k\le M}$ then for any $k\le M$, one has

  $$\begin{array}{c}\hfill \underset{N\to \infty }{lim}\mathbb{E}{\langle m_N\left(\mathit{\sigma }\right|{\mathit{\xi }}^k)\rangle }_N=\frac{\partial }{\partial {\lambda }_k}p(\mu ,\nu ,\lambda )={\overline{x}}_k\end{array}$$

  (57)
  and

  $$\begin{array}{c}\hfill \underset{N\to \infty }{lim}\mathbb{E}{\langle q_{12}^2\rangle }_N=1-4\frac{\partial }{\partial \mu }p(\mu ,\nu ,\lambda )=\int q^{2}d{\chi }^{\ast }\left(q\right).\end{array}$$

  (58)
  where ${\chi }^{\ast }\left(q\right)$ denotes the unique measure solving the Parisi variational principle in Theorem 1.
• If $\left\{{\left|x\right|}^2,x\in {\Omega }_{\mu ,\nu ,\lambda }\right\}=\left\{\right|\overline{x}{|}^2\}$ is a singleton then

  $$\begin{array}{c}\hfill \frac{\partial }{\partial \nu }p(\mu ,\nu ,\lambda )=\frac{|\overline{x}{|}^2}{2}.\end{array}$$

  (59)

More generally, let y be one of the variables $\sqrt{\mu },\nu ,{\lambda }_1,\dots ,{\lambda }_M$, then the function $y\mapsto p(\mu ,\nu ,\lambda )$ is convex. By Danskin theorem (see [56]), $y\mapsto p(\mu ,\nu ,\lambda )$ is differentiable if and only if the set $\left\{{\displaystyle \frac{\partial \phi (x;\mu ,\nu ,\lambda )}{\partial y}},x\in {\Omega }_{\mu ,\nu ,\lambda }\right\}$ is a singleton.

5. Conclusions and Perspectives

In this paper, we offer an overview of the Parisi formula from a mathematical physics perspective, emphasizing its potential applications, particularly in addressing the mismatched inference problem outlined earlier. Building upon our previous work [12], we investigate a scenario where a statistician, tasked with reconstructing a finite-rank matrix, lacks knowledge about the underlying matrix generation process, including both the matrix elements and its rank. We consider the case in which the statistician assumes a rank-one matrix, leading to a mismatch between the "true" Bayes posterior and the one used for inference. Our key contribution is the proof that, contrary to what happens in the Bayes-optimal setting, this Bayesian mismatch induces replica symmetry breaking in the model. Consequently, we express the pressure of the corresponding spin glass as an infinite-dimensional variational principle over the space of distributions on [0, 1].

The chosen mismatch scenario shares some similarities with those studied in [57,58] with the fundamental difference being that here the rank of the hidden matrix is finite. In a recent work [59], the authors consider a general case of mismatch, which includes mismatching priors and likelihoods. The mentioned paper proves a universality property with respect to the likelihood assumed by the statistician provided that observations remain independent given the ground truth.

Despite these advancements, all the proofs available so far in the literature break down when considering a high-rank hidden matrix. To rigorously comprehend this scenario, addressing the solution of the Hopfield model is of crucial importance. However, to the best of our knowledge, its complete solution remains elusive [5,6].