Neural Activity in Quarks Language: Lattice Field Theory for a Network of Real Neurons

Abstract

Brain–computer interfaces have seen extraordinary surges in developments in recent years, and a significant discrepancy now exists between the abundance of available data and the limited headway made in achieving a unified theoretical framework. This discrepancy becomes particularly pronounced when examining the collective neural activity at the micro and meso scale, where a coherent formalization that adequately describes neural interactions is still lacking. Here, we introduce a mathematical framework to analyze systems of natural neurons and interpret the related empirical observations in terms of lattice field theory, an established paradigm from theoretical particle physics and statistical mechanics. Our methods are tailored to interpret data from chronic neural interfaces, especially spike rasters from measurements of single neuron activity, and generalize the maximum entropy model for neural networks so that the time evolution of the system is also taken into account. This is obtained by bridging particle physics and neuroscience, paving the way for particle physics-inspired models of the neocortex.

1. Introduction

Integrating observations of neural activity into a coherent theoretical framework is still a challenging task due to the volume and diversity of the experimental data. High-resolution recording techniques allow for simultaneous sampling of hundreds of neurons [1,2,3,4,5]. Although current probes for in vivo experiments usually record only a fraction of active neurons, the next generation promises to greatly increase this crucial parameter. Concerning the time scanning rate, interface performances exceeded the typical timescale of the neuron’s activity long ago, and a gigantic amount of data has been accumulated and published, along with a variety of methods and theories [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]. While some progress has been made in large-scale modeling, micro- and meso-scale models lag behind [3,24,25,26,27,28,29,30,31,32,33,34,35,36,37]. For better or worse, the situation resembles the “zoo of particle physics” prior to the introduction of the Standard Model. In this paper we introduce a lattice field theory (LFT) [21,23,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56] that is tailored to interpret data from multisite brain–computer interfaces (BCIs) in a systematic and physically grounded way that links the microscopic parameters to the experimental observations through well-known renormalization procedures. In short, LFTs discretize the space–time into a lattice grid and are commonly used in theoretical particle physics to facilitate numerical simulations and intractable calculations. Our starting point will be from the novel “kernel” [56] approach to LFTs, in part because of its simplicity and in part because it allows a natural connection with the theory of spin glasses [56,57,58,59,60,61], which has been proposed as model of neural activity [19,22,62,63,64,65,66] and for pattern storage in memory and learning [15,16,18,67,68]. We will assume that time evolution can be characterized by a discrete non-relativistic process of interacting binary fields, or Qubits [53,54,55], that from a neuroscience point of view can be interpreted as a field theoretic version of the Free Energy principle of the Bayesian Brain Theory (see for example Friston et al. [49,69,70]). We fully develop the formalism and the basic principles for binary raster diagrams (although the arguments can be readily extended to any Potts-like model with multi-spin interactions). The scope of this paper is to present the theory in full mathematical detail, so that it can be of reference in a wide range of settings from single neuron recordings to multi-layer perceptron networks and quantum Turing machines.

We will describe in detail the whole process of constructing, testing and interpreting actual experimental BCI observables from very basic theoretical principles. Our goal is to describe the microscopic support of cognitive processes, and our fundamental assumption is that it can be exactly encoded into a digital quantum network (qubit network), emergent from the on–off states of the action potentials (digital neurons hypothesis). We base this on the fact that neurons are distinct objects, and that they have a refractory period ensuring regularity in the time domain. Then, we try to take advantage of the presumed information “bottleneck” offered by the digital neuron hypothesis to simplify the mathematical analysis, and eventually bridge neuroscience, physics and data science within the formalism of quantum field theory. The quantum formalism may look redundant at first, as classical evolution is always recovered as a subcase of quantum evolution, but we believe that it will eventually be proven crucial to describe the neural correlates of cognitive processes.

How to Read This Paper

This work is intended to provide a comprehensive exploration of the themes at hand. For a shorter and more readable synopsis please refer to [71]. The paper is organized into four main sections: the introduction (Section 1) provides a preview of the main results (Section 2), a basic theoretical section introducing the observables (Section 3) and another more technical section to develop the neural LFT in full detail (Section 4). In the final section (Section 5), we apply the theory to real experimental situations. The length and organization of this paper are probably not optimal for readers who are unfamiliar with statistical mechanics and computational neuroscience. Given the conceptual span, multidisciplinary nature and novelty of the framework, there will unavoidably be parts that are more difficult to read or that may appear trivial, depending on the background of the reader. Nonetheless, we believe that both physicists and neuroscientists can read and understand this paper with a similar level of effort. Achieving balanced “dissatisfaction” between physics and neuroscience readers is an important goal of this paper. Indeed, although previous proposals for neural LFTs exist [21,23,49,51,52,72], in our opinion, none tackled the problem of linking with the actual experimental observables in a way that can also be managed by non-physicists.

2. Main Results

2.1. Neural Activity in Terms of Lattice Field Theory

It is widely accepted that the most basic computational units of the brain are the neurons. Neurons receive electro-chemical inputs from other neurons through dendrites, which are then integrated into the cell body. When the integration reaches a threshold, the neuron generates electrical impulses called action potentials, or spikes. If such a threshold is not reached, no spike is generated. When recording neural activity, e.g., during a neurophysiology experiment, it is usual to collect the timing and occurrence of spikes of an arbitrary number N of individual neurons and align them, in an arbitrary time window T, with events or stimuli specific to the chosen experimental paradigm. The matrix with N rows (neurons) and T columns (time) that encodes this information is called a spike raster. Calling V the space and S the time in which our system “lives”, we note that V is regularized by the intrinsic discretization of its units, i.e., the neurons are discrete objects, and that S is regularized by the physiological existence of an absolute refractory period and fixed by the natural temporal ordering of the observed dynamical evolution. This implies that when studying an ensemble of neurons, we can represent space–time with a set of discrete points or sites of a lattice. Formally, we define a spatial mapping onto the following ordered set of vertices (see Section 3.1),

$$V:=\left\{1\le i\le N\right\},S:=\left\{1\le \alpha \le T\right\}$$

(1)

where the time window is regularized into sub-intervals $\alpha$ according to a hypothetical “clock time” $\tau$, corresponding to the minimum time between two computational operations of the neuron. Considering that, after a spike, a neuron enters a refractory period during which it is temporarily unable to generate another one, as $\tau$ it would be natural to consider charge time + discharge time + absolute refractory period + relative refractory period. However, since $\tau$ may vary significantly depending on the experimental setting, for an accurate digitalization of the signal, it is convenient to consider its smallest possible value, i.e., the typical duration of a spike: $\tau \approx 1$ ms. Within a $\tau$, the neuronal computational unit i can be either silent or active, which can be represented by a binary variable. The raster (or kernel) $\mathsf{\Omega }$ can be explicitly written as:

$$\mathsf{\Omega }:=\{{\phi }_i^{\alpha }\in \{0,1\}:i\in V,\alpha \in S\}$$

(2)

in which ${\phi }_i^{\alpha }$ is the binary variable representing the activity of the i-th neuron at time $\alpha$. After these simple observations, one recognizes that $\mathsf{\Omega }$ naturally provides all the necessary information to describe the observed neural dynamic. From now on we will refer to $\mathsf{\Omega }$ as the (neural) activity kernel, or simply, kernel [56]. The neural dynamic is expected to follow some causal evolution influenced by the prior states, i.e., a dynamical process with memory. As already argued by several authors [21,23,49,51,52,72], it is reasonable to assume that such dynamics can be described by a quantum evolution, so that the formalism of quantum field theory [21,23,38,40,41,42,43,44,45,46,51,52,53,54,55,56] can be applied: this is a harmless assumption, since classical evolution can always be retrieved as a sub-case of quantum evolution. Then, let us assume that the evolution of ${\phi }_i^{\alpha }$ in $\alpha$ can be characterized by a discrete process of interacting binary fields, or Qubits [53,54,55,56]. We can model the time evolution of ${\phi }_i^{\alpha }$ by considering its statistical mechanics counterpart on a lattice [21,23,38,40,41,42,43,44,45,46,51,52,53,54,56]: to do so, we only need to formally define a few quantities, familiar to neuroscientists, that can be obtained from the binary kernel $\mathsf{\Omega }$. Just like a system of particles, we can describe how the N neurons, represented by $\mathsf{\Omega }$, interact with their surroundings in discrete lattice space–time with a single expression enclosing static and dynamic properties. In short, we postulate the existence of the lattice action

$$\mathcal{A}:{\{0,1\}}^{VS}\to \mathbb{R},$$

(3)

that allows us to derive the Lagrangian description of the system and, through the principle of least action [47,73], the corresponding statistical theory. Hence, let $\mathcal{O}$ be a test function of $\mathsf{\Omega }$, we denote the ensemble average with respect to $\mathcal{A}$ with angle brackets and formally define it as follows (WGL formula/Gibbs average/softmax average):

$$\langle \mathcal{O}\left(\mathsf{\Omega }\right)\rangle :=\sum _{\mathsf{\Omega }\in {\{0,1\}}^{VS}}\mathcal{O}\left(\mathsf{\Omega }\right)\frac{\exp [-\lambda \mathcal{A}(\mathsf{\Omega }\left)\right]}{{\sum }_{{\mathsf{\Omega }}^{\prime }\in {\{0,1\}}^{VS}}\exp [-\lambda \mathcal{A}\left({\mathsf{\Omega }}^{\prime }\right)]}.$$

(4)

The classical (non-quantum) limit of the theory is obtained taking the limit of infinite $\lambda$, corresponding to a zero Planck constant, or the zero temperature limit of canonical statistical mechanics [74]. Let us introduce $\mathsf{\Phi }$ the space correlation matrix and $\mathsf{\Pi }$ the time correlation matrix (joint-spike matrix, JS),

$$\begin{array}{cc}\hfill & \mathsf{\Phi }:=\{{\varphi }_{ij}\in \left[0,1\right]:i,j\in V\},{\varphi }_{ij}:=\frac{1}{T}\sum _{\alpha \in S}{\phi }_i^{\alpha }{\phi }_j^{\alpha }\\ \hfill & \mathsf{\Pi }:=\{p^{\alpha \beta }\in \left[0,1\right]:\alpha ,\beta \in S\},p^{\alpha \beta }:=\frac{1}{N}\sum _{i\in V}{\phi }_i^{\alpha }{\phi }_i^{\beta }\end{array}$$

(5)

Indicating the transpose operation with the symbol †, these are straightforwardly obtained from the kernel through the relations

$$\mathsf{\Omega }{\mathsf{\Omega }}^{†}/T=\mathsf{\Phi },{\mathsf{\Omega }}^{†}\mathsf{\Omega }/N=\mathsf{\Pi },$$

(6)

which are often used automatically (and unconsciously) to calculate spatial and temporal correlations between experimental data. Combining these quantities, we can write a simplified action

$$\mathcal{A}\left(\mathsf{\Omega }|A,B,I\right):=T\sum _{i\in V}\sum _{j\in V}{A}_{ij}{\varphi }_{ij}+N\sum _{\alpha \in S}\sum _{\beta \in S}{B}^{\alpha \beta }{p}^{\alpha \beta }+\sum _{i\in V}\sum _{\alpha \in S}{I}_i^{\alpha }{\phi }_i^{\alpha }.$$

(7)

where the matrix A of potential interactions and the matrix B of kinetic interactions control the theory and I is the input kernel that collects the external influences. The full derivation of Equation (7), omitted here for the sake of conciseness, can be found in Section 4. This is our first main result: an explicit expression for the action of a network of real neurons that depends on easily accessible experimental quantities. Although commonly used to derive and comment on empirical results in neuroscience research [75,76], the empirical correlation matrices were, hitherto, not related to each other or ascribable to a precise physical meaning. The action $\mathcal{A}$ provides a recipe for interpreting them in a physically grounded way, setting them within a general theoretical framework that portrays the dynamics of a system in terms of kinetic and potential energies. This entails being able to put a plethora of experimental results under one theoretical hat, using a coherent physical theory. The ingredients of the recipe are the three observables $\mathsf{\Phi }$, $\mathsf{\Pi }$ and $\mathsf{\Omega }$, that encode all the information about the system, the parameters of the theory A and B that control the fluctuations and the boundary conditions I. We will call these triplets the hypermatrix and the inverse hypermatrix, respectively, since each group of observables can be arranged into a single matrix as in Figure 1 and Figure 2. Many properties of the inverse matrices can be inferred by simple self-consistency conditions: for example, causality implies that B is an upper triangular (see Section 4.1.3). Our method represent a natural generalization of the maximum entropy principle proposed in the works of Schneidman, Tkacik and colleagues [62,63,64,65,66] where an Ising model with variable couplings is used to fit ex vivo recordings of a salamander retina (see Section 2.2, Section 2.3 and Section 4.1.2). Moreover, it is remarkable that the Principal Component Analysis (PCA) and the maximum entropy principle can be linked in a natural way within the proposed LFT context. The PCA is probably the most used numerical method in many scientific fields and, in neuroscience, a large amount of data from decades of research are already available in this form to feed machine learning methods [24,35,77,78,79,80,81,82]). Given the operator relations between the kernel and the correlation matrices, it can be shown (Section 2.4) that both PCA in the space domain and the maximum entropy principle are special cases of the proposed LFT with zero kinetic terms ($B=0$), while the action associated with a PCA in the time domain is that of a purely kinetic LFT ($A=0$), i.e., the dual of the maximum entropy principle in the so-called momentum space. Although in this paper we will stick to a strictly real-space analysis, we notice that there are also many powerful spectral methods that can be used to analyze rectangular arrays, like the singular value decomposition (SVD). For example, using the SVD, one finds that the spectrum of $\mathsf{\Phi }$ and $\mathsf{\Pi }$ is the same apart from a scaling factor. This last fact has its own physical importance and will be discussed elsewhere.

2.2. Generalization of the Maximum Entropy Principle

We show that our LFT is a generalization of the maximum entropy principle as presented in the work of Schneidman, Tackcik and others [62,63,64,65,66]. With some algebra and a global rescaling of the quantities (see Section 4.1.2) we can write the action in the magnetic representation:

$$\mathcal{A}\left(M|A,B,h\right)=\sum _{i\in V}\sum _{\alpha \in S}{h}_i^{\alpha }{\sigma }_i^{\alpha }+\frac{T}{4}\sum _{i\in V}\sum _{j\in V}{A}_{ij}{c}_{ij}+\frac{N}{4}\sum _{\alpha \in S}\sum _{\beta \in S}{B}^{\alpha \beta }{q}^{\alpha \beta }.$$

(8)

where h is linearly related to I (see Section 4.1.2). In the spin case the hypermatrix will therefore consist of M, C and Q. By expanding the definition of $c_{ij}$ we immediately note that in the limit $B\to 0$ we have:

$$\mathcal{A}\left(M|A,0,h\right)=\sum _{\alpha \in S}\sum _{i\in V}{h}_i^{\alpha }{\sigma }_i^{\alpha }+\frac{1}{4}\sum _{\alpha \in S}\sum _{i\in V}\sum _{j\in V}{A}_{ij}{\sigma }_i^{\alpha }{\sigma }_j^{\alpha }.$$

(9)

In this way, a replicated version [60] of the Hamiltonian (i.e., the total energy of the system) used in the works of Schneidman and Tkacik is obtained. Notice that the limit $T\to 1$, corresponding to a single replica of the system, is exactly equivalent to the max entropy model [62,63,64]

$$\mathcal{A}\left(M|A,0,h\right)=\sum _{i\in V}{h}_i^1{\sigma }_i^1+\frac{1}{4}\sum _{i\in V}\sum _{j\in V}{A}_{ij}{\sigma }_i^1{\sigma }_j^1.$$

(10)

Thus, the max entropy principle is recovered as specific case of a field theory with zero kinetic energy. For what we have shown, the authors approximate the activity with a LFT at equilibrium with $B=0$, which corresponds to a field theory with zero kinetic energy.

2.3. Scaling Test for Axonal Connectivity

Let us now show a simple possible application to neural recordings, considering the case of the salamander retina dataset shown in the papers of Schneidman, Tkacik and colleagues [62,63,64] (Figure 3 and Figure 4 ). Remarkably, they were able to reconstruct the A matrix for small groups of neurons in a salamander retina, thus obtaining both the correlation matrix and its dual in the parameter space. In Figure 1f of their paper [63] they show the distributions of the reconstructed couplings ${\tilde{J}}_{ij}$ for some values of N. The distributions of ${\tilde{J}}_{ij}$ are indeed approximately Gaussian and it would be very interesting to verify the scaling of the variance of such distributions at various N. Concerning the space couplings, for mean field models [56,57,58] the pairwise interaction is the sum of two terms (stationary and fluctuating)

$$A_{ij}={\tilde{J}}_{ij}+\delta {\tilde{J}}_{ij},$$

(11)

and hence, for the thermodynamic limit to exist, it is necessary that the $h_i$ be of order $O\left(1\right)$ in the number of neurons and that ${\tilde{J}}_{ij}$ and $\delta {\tilde{J}}_{ij}$ scale correctly. This depends on the connectivity of the matrix of axonal adjacencies (axon matrix):

$$\mathsf{\Lambda }=\left\{{\mathsf{\Lambda }}_{ij}\in \left\{0,1\right\}:i,j\in V\right\}.$$

(12)

Average connectivity is defined as follows:

$$g\left(\mathsf{\Lambda }\right):=\frac{1}{N}\sum _{i\in V}\sum _{j\in V}{\mathsf{\Lambda }}_{ij}.$$

(13)

Therefore, to normalize correctly, one must take

$${\tilde{J}}_{ij}=\frac{1}{g\left(\mathsf{\Lambda }\right)}{J}_0{\mathsf{\Lambda }}_{ij},\delta {\tilde{J}}_{ij}=\frac{1}{\sqrt{g\left(\mathsf{\Lambda }\right)}}{J}_{ij}{\mathsf{\Lambda }}_{ij}$$

(14)

with $J_0$ and $J_{ij}$ Gaussian variables of unit variance and zero mean. For fully connected models, we have $g\left(1\right)=N$ and then

$${\tilde{J}}_{ij}=\frac{1}{N}{J}_0,\delta {\tilde{J}}_{ij}=\frac{1}{\sqrt{N}}{J}_{ij}$$

(15)

For models with large connectivity but that are sub-linear in the number of neurons, one can consider $g\left(\mathsf{\Lambda }\right)=N^{\alpha }$ with $0<\alpha <1$,

$${\tilde{J}}_{ij}=\frac{1}{N^{\alpha }}{J}_0{\mathsf{\Lambda }}_{ij},\delta {\tilde{J}}_{ij}=\frac{1}{\sqrt{N^{\alpha }}}{J}_{ij}{\mathsf{\Lambda }}_{ij}$$

(16)

whereas for finite-dimensional models we have that $g\left(\mathsf{\Lambda }\right)=O\left(1\right)$. From preliminary analysis (we extracted data from Figure 1f of Tkacik et al., 2009 [63] with G3data and performed a Gaussian fit to find the variances, Figure 4), we confirm that the couplings are approximately Gaussian, but the scaling exponent appears to be $\alpha =1/2$ and not $\alpha =1$ as for the Sherrington–Kirkpatrick model [56,57,58,60,87]. This would be interesting, since the system would admit a thermodynamic limit and still have sufficient connectivity to manage the data with a mean-field theory [56,57,58,60,87]. Also, it would be very interesting to see the full hypermatrix to which the covariance matrix in Figure 1 of Tkacik et al. [63] belongs, and even more interesting would be to fit such a hypermatrix with the field theory presented in this paper.

Figure 3. Example of Columnar organization of retina and the decimation procedure from $x,y$ to $x^{\prime },y^{\prime }$. Diagram elaborated from Figure 2 of Jones 2000 [88]. We re-scaled the vertical dimension z for improved visualization.

Figure 3. Example of Columnar organization of retina and the decimation procedure from $x,y$ to $x^{\prime },y^{\prime }$. Diagram elaborated from Figure 2 of Jones 2000 [88]. We re-scaled the vertical dimension z for improved visualization.

Figure 4. We extracted the distributions of the reconstructed couplings $J_{ij}$ from Figure 1f of Tkacik et al., 2009 [63] with G3data (A) and computed the scaling of the parameters, first from Gaussian fits (B,C) and then from the first two moments of the distributions (B–D). Both methods confirm that the couplings are approximately Gaussian with scaling exponent $\alpha =1/2$. We remark that finding the parameters of the theory using scaling techniques like this is typical of LFT analysis of elementary particle theory, where they are typically used to estimate particle masses and other observables.

Figure 4. We extracted the distributions of the reconstructed couplings $J_{ij}$ from Figure 1f of Tkacik et al., 2009 [63] with G3data (A) and computed the scaling of the parameters, first from Gaussian fits (B,C) and then from the first two moments of the distributions (B–D). Both methods confirm that the couplings are approximately Gaussian with scaling exponent $\alpha =1/2$. We remark that finding the parameters of the theory using scaling techniques like this is typical of LFT analysis of elementary particle theory, where they are typically used to estimate particle masses and other observables.

2.4. Relation with the Principal Component Analysis

Here we show that the PCA can also be interpreted as a special case of our LFT. In particular, the PCA can be understood as projecting the data into a subspace such that the projected data have a minimum discrepancy with the original one. In the following, we explicitly consider only the projection into the spatial domain, but the temporal projection is similar. Then, let $V^{\prime }$ be a subset of V with size $n<N$,

$$X:=\{x_i^k\in \mathbb{R}:i\in V,k\in V^{\prime }\},$$

(17)

being a real valued kernel with N rows and n columns, and let us introduce the set of kernels with orthonormal columns (in this paragraph we denote by $\mathbb{I}$ the identity matrix)

$$\mathcal{T}:=\{X\in {\mathbb{R}}^{V{V}^{\prime }}:X^{†}X=\mathbb{I}\}.$$

(18)

The columns span a n-dimensional subspace of ${\mathbb{R}}^N$, and the projection of $\mathsf{\Omega }$ into this subspace is $X{X}^{†}\mathsf{\Omega }$. The PCA aims to identify the subspace such that the discrepancy between the operators $X{X}^{†}\mathsf{\Omega }$ and $\mathsf{\Omega }$ is as small as possible in the Frobenius norm. The objective function is

$$\mathcal{A}\left(\mathsf{\Omega }\right|X):= \parallel X{X}^{†}{\mathsf{\Omega }-\mathsf{\Omega }\parallel }_F^2=\mathrm{Tr}({\mathsf{\Omega }}^{†}\mathsf{\Omega }-X^{†}\mathsf{\Omega }{\mathsf{\Omega }}^{†}X)$$

(19)

where ${\parallel ·\parallel }_F$ denotes the Frobenius norm and where we applied ${\parallel D\parallel }_F^2=\mathrm{Tr}\left(D{D}^{†}\right)$, with $\mathrm{Tr}(·)$, indicating the trace of the operator. This and the othonormal constraint constitute a constrained optimization problem: calling Y the solution to this problem, we write

$$\mathcal{A}\left(\mathsf{\Omega }\right|Y)=\underset{X\in \mathcal{T}}{min}\mathcal{A}\left(\mathsf{\Omega }\right|X)=\underset{X\in \mathcal{T}}{min}\mathrm{Tr}({\mathsf{\Omega }}^{†}\mathsf{\Omega }-X^{†}\mathsf{\Omega }{\mathsf{\Omega }}^{†}X).$$

(20)

The minimum is found by choosing Y as the n largest eigenvector of $\mathsf{\Omega }{\mathsf{\Omega }}^{†}$ (or the largest left-singular vectors of $\mathsf{\Omega }$), see Goodfellow et al. [89] for a proof. In the end, one finds

$$\mathcal{A}\left(\mathsf{\Omega }\right|Y)=-\mathrm{Tr}\left(Y^{†}\mathsf{\Omega }{\mathsf{\Omega }}^{†}Y\right)=-T\sum _{i\in V}\sum _{j\in V}{\varphi }_{ij}\sum _{k\in V^{\prime }}{y}_i^k{y}_j^k.$$

(21)

We conclude that the action of PCA in the space domain is that of a LFT with $B=0$ and

$$A_{ij}=-\sum _{k\in V^{\prime }}{y}_i^k{y}_j^k,$$

(22)

and is therefore a special form of the maximum entropy principle described above.

2.5. A Model for Cortical Recordings

The LFT formalism makes it possible to compare cortical recordings of neural activity with a renormalized field theory. The most used interface to simultaneously record collective neural activity is the silicon-based multilectrode array Utah 96 (Blackrock Microsystems, Salt Lake City, UT, USA) [3,4,34,90]. To date, there are about 20 years of recordings of neural activity made with Utah 96 across different species and under hundreds of different experimental conditions, with thousands of kernels already available. The Utah array is a square grid with a 10 × 10 electrode arrangement with a total of 96 channels (the vertices of the square have no record). Due to its planar geometry, the length of its electrodes (around 1.5 mm penetration into the cortex) and their pitch (40 μm), the Utah 96 is able to record from neurons belonging to horizontally separated cortical assemblies sampled from the same superficial cortical layer z (an example is given in Figure 1 panel A). We will refer to such assemblies as minitubes [83,84,88,91,92,93,94,95,96,97,98,99] (see Figure 3 for a sketch). As a comparison, a multi-electrode array with a linear geometry, like a single shank with multiple contact points arranged in a vertical fashion (e.g., like Neuropixels [2,100] or SiNAPS [1] probes), would sample from neurons across various layers within a single minitube [84,94]. Given that each electrode of the Utah array is designed to approximately record the activity of individual minitubes at a distance enough to avoid self-interaction terms, we can model any of its recording as a decimated minitube lattice and the dynamic evolving around each electrode tip with an on/off field ${\widehat{\phi }}_{xy}^{\alpha }$ that identifies the state of the observed minitube (see Section 4.2.6). Let us name $xyz$ as the coordinates of a lattice such that z represents the average height from the surface of the cortex at which a given layer is located. $V_{xyz}$ represents the volume occupied by (all) the neurons present there and $xy$ is the position of the minitube section in the horizontal plane. Each cortical layer z has its kernel

$${\mathsf{\Omega }}_z:=\{{\mathsf{\Omega }}_{xyz}^{\alpha }\in {\left\{0,1\right\}}^{V_{xyz}}:xy\in {\mathbb{L}}_2,\alpha \in S\},$$

(23)

where ${\mathbb{L}}_2$ is a two-dimensional lattice with an average lattice step around the diameter of the individual minitube. Calling $\mathbb{I}(·)$ the indicator function, we can now define ${\widehat{\phi }}_{xy}^{\alpha }$ as:

$${\widehat{\phi }}_{xy}^{\alpha }:=\mathbb{I}({\mathsf{\Omega }}_{xyz}^{\alpha }\ne 0)$$

(24)

which corresponds to assuming that the activation of any one of the neurons in the cell $V_{xyz}$ corresponds to the activation of the entire cell and with a high probability of the activation of the whole minitube. Next, to model the spacing between the probing contacts of the array we apply renormalization considering a decimated lattice $x^{\prime }{y}^{\prime }$ whose step is much greater than the diameter of the individual minitube. One way of renormalizing is the so-called renormalization by decimation [101,102,103], in which the details of the system at small scales are systematically simplified by integrating out most of the degrees of freedom (spins in magnetic systems, neurons in this context). In our case, it corresponds to leaving only those on the decimated lattice $x^{\prime }{y}^{\prime }$ at height z. This leads to the decimated activity kernel:

$$\widehat{\mathsf{\Omega }}:=\{{\widehat{\phi }}_{x^{\prime }{y}^{\prime }}^{\alpha }\in \left\{0,1\right\}:x^{\prime }{y}^{\prime }\in {\mathbb{L}}_2^{\prime },\alpha \in S\},{\widehat{\phi }}_{x^{\prime }{y}^{\prime }}^{\alpha }:=\mathbb{I}({\mathsf{\Omega }}_{x^{\prime }{y}^{\prime }z}^{\alpha }\ne 0)$$

(25)

Since $\widehat{\mathsf{\Omega }}$ describes the on/off activity recorded by each electrode tip, we call it the “electrode” kernel (the full derivation is in Section 4.2.6 and Section 5.1). $\widehat{\mathsf{\Omega }}$ yields the empirical hypermatrix shown in Figure 1 panel D. The space arrangement of $\widehat{\mathsf{\Omega }}$ for square multielectrode arrays like the Utah 96 is shown in Figure 1 panel B. Explaining how a cortical recording modeled in these terms is comparable to a renormalized field theory is our second main result. In Section 4.2.6, we show explicitly a very simple example of how to compute corrections for the effective theory in the semi-classical limit (near-zero Planck constant) [56]. It should be noted that more accurate renormalization schemes could be achieved by many other established methods [52,57,101,102,103,104,105,106,107]. In general, the exact renormalized theory for a generic cortical recording will depend on the details of the system, the recording interface, the experimental settings and other features that should be considered case by case. Indeed, modeling and calculating the non-linear corrections for the effective theory would be one of the core aspects on which a transfer of expertise from nuclear physics and statistical mechanics to neuroscience could be crucial. In the case of Utah 96, the interface appears to be designed to take individual minitubes for each electrode at a distance enough to avoid self-interaction terms, so we can assume that, apart from systematic errors, sensor degradation, etc., the data can be identified with a decimated version of the kernel of columnar activities, i.e., with ${\widehat{\mathsf{\Omega }}}_{x^{\prime }{y}^{\prime }}$ defined before. It would be extremely interesting to reconstruct the couplings from an experimental hypermatrix, e.g., of the motor control experiments presented in Pani et al. [35] (see Figure 1 and Figure 2). However, this is a task with a significant computational burden: to reduce the number of computations required, one could eventually bin or decimate the kernel on a larger clock time. Anyway, we remark that even by looking at the hypermatrix alone, in particular at the kernel and the overlap matrix, it is already possible to appreciate most of the results of Pani et al. [35] without resorting to numerical methods such as PCA (that can, however, be interpreted in this framework). In previous works [35,82,86,108,109,110] the modulation of the activity is observed between the Go and movement onset (M_on), which is called a “motor plan”. We can also appreciate a stationary rhythmic activity before the Go signal (see Figure 1 and Figure 2) and after the M_on revealed by transverse waves in the joint spike matrix $\mathsf{\Pi }$ that is not detected with standard methods (e.g., PCA) and that is suggestive of a time crystal [111,112]. A detailed discussion of the experiment shown in Figure 1 can be found in Section 5.2.

2.6. Conclusions

In conclusion, we showed that applying lattice methods from elementary particle theory to natural neurons should be possible and fruitful, but the understanding of this theoretical framework will still require substantial work. However, given the advanced development of LFTs and their vast range of applicability, knowledge exchange with neuroscience would be beneficial for the theoretical development of the latter in the near future (see Section 5.3.1 and Section 5.3.2) and for both in the long run (see Section 5.3.3). The encounter opens exciting avenues for interdisciplinary research, facilitating connections between computational neuroscience and other fields of physics that utilize LFTs. Indeed, LFTs became of crucial importance far beyond their traditional realms, encompassing cellular automata [113,114,115], number theory [53] and computational modeling [61,116,117,118]. Thus, it is reasonable to think that they could do the same in neuroscience once the appropriate common theoretical frame is established [21,23,49,52,56]. With our manipulations we showed that determining the effective theory describing the dynamics of assemblies of neurons in the neocortex is possible in terms of LFT, at least in our formalism [57]. Moreover, given its spatial symmetries [83,84,88,91,92,93,94,95,96,97,98,99], it is possible that the topology of such a theory is either mean-field or two-dimensional, with the cortical layers behaving as interacting fields just as in elementary particle theory. Rethinking neural interactions in this manner could significantly simplify the analytical construction of effective theories. This is because, whether in the mean field [52,56,57,58,106,107] or dimension 2 [101,102,103,104,105,119], established schemes for analyzing, simulating, renormalizing and, in some cases, exactly solving such theories already exist. In truth, it is not possible to predict what a collision between particle physics and neuroscience might lead to, yet the arguments in this work strongly suggest that it would be worth finding out.

3. Theoretical Methods

3.1. Fundamentals

Here we introduce some fundamental notations to describe a patch of cortical tissue as a dynamical system on lattices. In these first sections we will consider variables that are aimed to model the whole part of the nervous system involved in the neural computation, of which the actually observed neurons (e.g., during a neurophysiology experiment) are typically a sparse subset. This part is structured as follows: in the first sections we introduce the notation and the basic quantities, the observables and their physical significance, various notions of ergodicity and the fundamental one of stationarity. A discrete Lagrangian description of the system is then introduced in terms of LFT and relations to observables and a general statistical theory are established. A simple kernel “renormalization” scheme, based on Franchini 2023 [56], is then introduced to deal with the problem of relating the microscopic theory with marginals on a sparse subset of neurons. Finally, some applications to neuroscience are described, such as the possibility of constructing an effective theory by renormalizing via decimation [102,103,104,105] of a two-dimensional lattice of cortical columns. For the detailed derivation of the Lattice Field Theory (LFT) in kernel formalism [56] and its associated statistical theory, please refer to Section 4 and Section 4.2.

3.1.1. Map on Vertex

Let N be the number of neurons involved in the task; these are arbitrarily mapped onto the ordered set of vertices

$$V:=\left\{1\le i\le N\right\}.$$

(26)

Indexing $i\in V$ is defined short of a two-way map that shuffles the index

$$\theta :V\to V,{\theta }^{-1}:V\to V$$

(27)

This map constitutes a parameter of the representation and ideally should be chosen so as to highlight the function of the various neurons observed during the task, i.e., highlighting any groups of neurons that belong to the same population, or computational structure.

3.1.2. Identify the $\theta$ Highlighting the Space-Time Structures

The neurons involved in the task (all of them, not just those actually observed) could be partitioned into further subgroups based on function, location, average activity and times when this varies. For example, it is possible to sort neurons by average activity or by the earliest time at which they significantly change their initial state. In the case of multi-electrode array interfaces, it is also possible to define a unique sorting of the experimental rasters based on the position of the sensor channels. Given the sampling rate of neural interfaces, it is assumed that the temporal sampling of the data is much finer than any functional level of interest, while the spatial resolution could be severely limited. This issue will be addressed in subsequent sections; for this one we assume that $\theta$ is arbitrary unless otherwise stated.

3.1.3. Dynamical System

Let us represent the electric field of the neurons with a semi-compact (i.e., still not discretized in the time variable) real valued kernel

$$X:=\left\{x_V\left(t\right)\in {\mathbb{R}}^N:t\in \left[0,T\right]\right\},x_V\left(t\right):=\left\{x_i\left(t\right)\in \mathbb{R}:i\in V\right\}$$

(28)

In general, we assume that the vector of the action potential $x_V$ follows a causal evolution

$$x_V\left(t\right)=f\left(x_V\left(s\right):s<t\right),$$

(29)

determined by the previous activity (dynamic process with memory) according to a hypothetical law f which in principle could be stochastic [69,70]. We further assume that such a law can be described or approximated by a quantum time evolution so that the formalism of statistical field theory [42] can be applied (notice that classical evolution is a subcase of quantum evolution).

3.1.4. Classical Lagrange Mechanics

Let us briefly recall the fundamental assumptions of the Lagrangian mechanics in a continuous time: we assume the existence of the “Lagrangian” function, canonically interpreted [120] as the difference between the kinetic and the potential energy of the system at given time. Then, the classical action is defined from the Lagrangian by integrating over the time

$$\mathcal{A}\left(X\right):={\int }_0^{T}dt\mathcal{L}\left[t,x_V\left(t\right),{\partial }_t{x}_V\left(t\right)\right],$$

(30)

we denoted the derivative with respect to time with ${\partial }_t:=d/dt$. Then, the associated evolution is a stationary point of the action, Y, usually a minimum. The equations of motion are also obtained from the Lagrangian through the celebrated Euler–Lagrange equations

$$\frac{\partial \mathcal{L}}{\partial x_i}={\partial }_t\left[\frac{\partial \mathcal{L}}{\partial \left({\partial }_t{x}_i\right)}\right]$$

(31)

that provide the stationary points of the action. On the other hand, determining whether a certain set of equations of motion

$${\mathsf{\Gamma }}_V\left(t,y_V,{\partial }_t{y}_V,{\partial }_t^2{y}_V\right)=0$$

(32)

admit a Lagrangian description is known as the inverse problem of Lagrangian mechanics. Many authors contributed to this topic and the necessary conditions, also known as the Helmoltz conditions (after Hermann Helmholtz),

$$\frac{\partial {\mathsf{\Gamma }}_i}{\partial \left({\partial }_t^2{y}_j\right)}=\frac{\partial {\mathsf{\Gamma }}_j}{\partial \left({\partial }_t^2{y}_i\right)},$$

(33)

$$\frac{\partial {\mathsf{\Gamma }}_i}{\partial y_j}-\frac{\partial {\mathsf{\Gamma }}_j}{\partial y_i}=\frac{1}{2}{\partial }_t\left[\frac{\partial {\mathsf{\Gamma }}_i}{\partial \left({\partial }_t{y}_j\right)}-\frac{\partial {\mathsf{\Gamma }}_j}{\partial \left({\partial }_t{y}_i\right)}\right],$$

(34)

$$\frac{\partial {\mathsf{\Gamma }}_i}{\partial \left({\partial }_t{y}_j\right)}+\frac{\partial {\mathsf{\Gamma }}_j}{\partial \left({\partial }_t{y}_i\right)}=2{\partial }_t\left[\frac{\partial {\mathsf{\Gamma }}_j}{\partial \left({\partial }_t^2{y}_i\right)}\right],$$

(35)

have been worked out in many general contexts [121,122]. The problems date back to Jacobi and has been attacked by many authors [122,123]. The lattice case has been studied by Crăciun and Opris [124], Bourdin and Cresson [125] and more recently by Gubbiotti [126,127]. Although this has been indirectly shown already for many important models of neural networks [49], a systematic test of these conditions would be an important test also for the quantum field theoretic description, which includes the free energy principle of Friston et al. [49,69,70].

3.1.5. Time Regularization

Since the space is already regularized by the natural discretization of the computational units (neurons/minitubes), in order to switch to the all-lattice system, it remains to regularize the time window. The window is therefore quantized in T sub-intervals of size $\tau$, in turn mapped onto the ordered set of vertices

$$S:=\left\{1\le \alpha \le T\right\}$$

(36)

preserving the temporal ordering. Notice that, in contrast to V, the map between time intervals and $\alpha \in S$ is naturally fixed by the time ordering of the process. The preservation of the time ordering will be a key feature of our field theoretic approach and is reminiscent of the time-ordered product of Green functions that is used as the starting point to derive the LSZ formula [128].

3.1.6. Multiscale Analysis

To study the system we will sometimes make use of the “multiscale” representation described at the end of Section 3 of [56] (Definitions 7 and 8, Figures 3.1 and 3.2). A workable definition requires the introduction of some degree of theoretical description, but in general it can be formalized with a joint partition of V and S in the computational cells, if any, of the various structures. Typically, structures at multiple scales, both in space (neurons) and time, will be identified. The most general partition is of the type described in Section 4, in which a sequence L of nested partitions (refinements) of the kernel S are the events and sub-events into which the data taken can be categorized, e.g., in the setting described in [35] we will have the single task level, which would be of the order of seconds, then the various sub-events (such as Go signal, Movement onset, etc.) on the 100 ms and finally the single actual “computations”, which might be isolated around 1–10 ms. Given the sampling rate of the most commonly used sensors, it is assumed that at the temporal level the data are scanned on a much finer scale than any functional level of interest, while the spatial resolution could be severely limited. This issue will be addressed in subsequent sections; for this one we assume that $\theta$ is arbitrary unless otherwise stated.

3.1.7. Clock Time

The time interval is quantized according to a hypothetical “clock” time $\tau$, corresponding to the time between two fundamental computational operations by the neuron. As described in the main text, it is convenient to consider $\tau$ = 1 ms.

3.1.8. Binary Computation Cell

Within a clock time the computation unit can be possibly active or silent: this can be encoded by a binary variable (digital neuron hypothesis),

$${\phi }_i^{\alpha }\in \left\{0,1\right\}$$

(37)

which is assumed to be the actual variable supporting the computation. The natural association is obviously with the active/silent state of the neuron during an interval equal to the clock time. This leads to the definition of the (neural) activity kernel $\mathsf{\Omega }$, i.e., a binary array with N rows and T columns that encodes the entire activity of the observed neurons within the time window. Formally, the kernel is a function of the type:

$$\mathsf{\Omega }:VS\to \left\{0,1\right\}$$

(38)

We can introduce the symbol ${\phi }_i^{\alpha }$ to represent the activity of the i-th neuron at time $\alpha$, in this way the kernel can be explicitly written as

$$\mathsf{\Omega }:=\left\{{\phi }_i^{\alpha }\in \left\{0,1\right\}:i\in V,\alpha \in S\right\}$$

(39)

The kernel is the general order parameter [56] and is assumed to contain the entire information needed to describe the observed process. Given the presence of an absolute refractory period, we expect that the regularized dynamical system

$${\phi }_V^{\alpha }=f({\phi }_V^{\beta }:\beta <\alpha )$$

(40)

is the one that naturally describes the evolution of the affected cortex sector and not the possible continuous theory that would be obtained by taking $\tau \to 0$.

3.1.9. Kernel of the Magnetizations

To transform from the binary representation into the spin system we use the map $\sigma =2\phi -1$ (which is equivalent to replacing the zeros by −1) that gives in the kernel of magnetizations:

$$M:VS\to \left\{-1,1\right\}$$

(41)

explicitly, the magnetization kernel is

$$M:=\left\{{\sigma }_i^{\alpha }\in \left\{-1,1\right\}:i\in V,\alpha \in S\right\}$$

(42)

The two descriptions are mathematically equivalent, but note that some observables may have different meanings. For example, the correlation between binary spikes is 1 only when neurons fire together and 0 otherwise, while the product between two spins is equal to 1 if the variables are equal, and negative if opposite. In practice, the first variable is sensitive only to the event in which the two neurons fire simultaneously, putting at the same rank (same null value) the events in which only one of the neurons fires and the one in which they are silent together. The second, on the other hand, distinguishes only whether or not they do the same thing, regardless of whether they fire or not.

3.2. Observables

In the following we define the observables of interest that can be obtained from the kernel.

3.2.1. Kernel Offset

The zero-order observable is the global average, or “offset”, of the kernel. This quantity is specific to the representation made with kernels and would be a kind of “ergodic” approximation of the activity in the space–time window that is considered. The offset is defined for both the activity and the magnetization kernels, respectively,

$$\overline{\mathsf{\Omega }}:=\frac{1}{T}\sum _{\alpha \in S}\frac{1}{N}\sum _{i\in V}{\phi }_i^{\alpha },\overline{M}:=\frac{1}{T}\sum _{\alpha \in S}\frac{1}{N}\sum _{i\in V}{\sigma }_i^{\alpha }$$

(43)

The two quantities are related by a linear relationship

$$\overline{M}=2\overline{\mathsf{\Omega }}-1$$

(44)

From a mathematical point of view they are proportional to the Grand Sum (sum of all the elements of a matrix). Such a kernel corresponds to the thermodynamic limit of a gas on a lattice with a particle density exactly equal to $\overline{\mathsf{\Omega }}$. For magnetic systems, in highly connected ones this can occur due to the presence of a constant external field, or from a uniform fully connected interaction of the Curie–Weiss type. We do not know if there are other magnetic or gaseous systems that have this property and are substantially different from magnetization eigenstates.

3.2.2. Row and Column Averages

At this point we can move to the description of the observables of order one. We define the row averages, which would be the average activity of the state at time $\alpha$ and the column averages, that is the average firing rate of the i-th neuron in the time window considered. The average activity is

$$\omega :=\left\{{\omega }^{\alpha }\in \left[0,1\right]:\alpha \in S\right\},{\omega }^{\alpha }:=\frac{1}{N}\sum _{i\in V}{\phi }_i^{\alpha },$$

(45)

with ${\omega }^{\alpha }$ space averages a time $\alpha$, while averages over the rows are

$$f:=\left\{f_i\in \left[0,1\right]:i\in V\right\},f_i:=\frac{1}{T}\sum _{\alpha \in S}{\phi }_i^{\alpha }.$$

(46)

where $f_i$ is the time-averaged activity of the i-th neuron in the time interval S. Notice that the average of the averages is in both cases equal to the offset

$$\frac{1}{N}\sum _{i\in V}{f}_i=\frac{1}{T}\sum _{\alpha \in S}{\omega }^{\alpha }=\overline{\mathsf{\Omega }}$$

(47)

Similarly, starting from the kernel of magnetizations we obtain the vector of space averages over the columns,

$$\mu :=\left\{{\mu }^{\alpha }\in \left[0,1\right]:\alpha \in S\right\},{\mu }^{\alpha }:=\frac{1}{N}\sum _{i\in V}{\sigma }_i^{\alpha },$$

(48)

the vector of time averages (on the rows)

$$m:=\left\{m_i\in \left[0,1\right]:i\in V\right\},m_i:=\frac{1}{T}\sum _{\alpha \in S}{\sigma }_i^{\alpha }.$$

(49)

where $m_i$ is the time average of the i-th spin in the time interval S. Again, the average of the averages is equal to the offset

$$\frac{1}{N}\sum _{i\in V}{m}_i=\frac{1}{T}\sum _{\alpha \in S}{\mu }^{\alpha }=\overline{M}$$

(50)

and between the averages made with spin and with spikes the same linear relationship holds: at this first level of description there are still no particular differences emerging between the two representations.

3.2.3. Spectra of the Averages (Quantiles)

We introduce the distributions (histograms) of the means

$$p_f\left(s\right):=\frac{1}{N}\sum _{i\in V}\delta \left(s-f_i\right),p_{\omega }\left(s\right):=\frac{1}{T}\sum _{\alpha \in S}\delta \left(s-{\omega }^{\alpha }\right)$$

(51)

and these two quantities are independent of the ordering of the index and allow for a comparison of the statistics of row and column averages. These can be expressed as cumulants

$$F_f\left(s\right):={\int }_0^{s}du{p}_f\left(u\right),F_{\omega }\left(s\right):={\int }_0^{s}du{p}_{\omega }\left(u\right)$$

(52)

alternatively, one can consider the “quantiles” $F_f^{-1}$ and $F_{\omega }^{-1}$ [58,129] that are the inverse functions of the cumulants. In the case of quantiles, it is particularly easy to construct the form of the function; let us use the notation $\left(i\right)$ and $\left(\alpha \right)$ for the maps that reorder the indices by increasing values (order statistics)

$$f_{\left(i\right)+1}\ge f_{\left(i\right)},{\omega }^{\left(\alpha \right)+1}\ge {\omega }^{\left(\alpha \right)},$$

(53)

the quantiles of f and $\omega$ are given by the following expressions:

$$\begin{array}{cc}\hfill & F_f^{-1}\left(s\right)=f_{\left(i\right)}\mathbb{I}\left(s\in \left[\left(i\right)/T,\left(\left(i\right)-1\right)/T\right]\right),\\ \hfill & F_{\omega }^{-1}\left(s\right)={\omega }^{\left(\alpha \right)}\mathbb{I}\left(s\in \left[\left(\alpha \right)/N,\left(\left(\alpha \right)-1\right)/N\right]\right).\end{array}$$

(54)

Magnetic versions of f and $\omega$ are defined in the same way. Given the linear relationship between the averages of the two representations, one can use the same index as that used for f and $\omega$ and write directly

$$\begin{array}{cc}\hfill & F_m^{-1}\left(s\right)=m_{\left(i\right)}\mathbb{I}\left(s\in \left[\left(i\right)/T,\left(\left(i\right)-1\right)/T\right]\right),\\ \hfill & F_{\mu }^{-1}\left(s\right)={\mu }^{\left(\alpha \right)}\mathbb{I}\left(s\in \left[\left(\alpha \right)/N,\left(\left(\alpha \right)-1\right)/N\right]\right).\end{array}$$

(55)

In what follows we may also refer to quantiles as “spectra” since for N and T finite are step functions whose support is naturally quantized in intervals of height $1/T$ for f, $1/N$ for $\omega$, $2/T$ for m and $2/N$ for $\mu$. This is not very relevant in the thermodynamic limit, but it is certainly relevant in many experimental situations: the discussion will be taken up later after introducing the Wasserstein distance.

3.2.4. Correlation Matrices

The variables of order two are the correlation matrices. It is possible to define a temporal correlation matrix, which in the case of the (neural) activity kernel would be the joint spikes matrix

$$\mathsf{\Phi }:=\{{\varphi }_{ij}\in \left[0,1\right]:i,j\in V\},{\varphi }_{ij}:=\frac{1}{T}\sum _{\alpha \in S}{\phi }_i^{\alpha }{\phi }_j^{\alpha }$$

(56)

and a spatial correlation matrix,

$$\mathsf{\Pi }:=\{p^{\alpha \beta }\in \left[0,1\right]:\alpha ,\beta \in S\},p^{\alpha \beta }:=\frac{1}{N}\sum _{i\in V}{\phi }_i^{\alpha }{\phi }_i^{\beta }$$

(57)

In general, these matrices are obtained from the kernel through the relations

$$\mathsf{\Omega }{\mathsf{\Omega }}^{†}/T=\mathsf{\Phi },{\mathsf{\Omega }}^{†}\mathsf{\Omega }/N=\mathsf{\Pi }$$

(58)

Note, however, that these observables are not completely independent from the averages and converge on the following matrices

$${\mathsf{\Phi }}_0:=\{f_i{f}_j\in \left[0,1\right]:i,j\in V\},{\mathsf{\Pi }}_0:=\{{\omega }^{\alpha }{\omega }^{\beta }\in \left[0,1\right]:\alpha ,\beta \in S\}$$

(59)

in the free-field approximation. We therefore define the connected correlation matrices, which describe the correlations between the fluctuations

$$C^{*}:=\mathsf{\Phi }-{\mathsf{\Phi }}_0,Q^{*}:=\mathsf{\Pi }-{\mathsf{\Pi }}_0$$

(60)

these are zero in the free-field limit and nontrivial in case the correlations between fluctuations are significant. Similarly to averages, we can also associate the correlations value distributions

$$p_{C^{*}}\left(s\right):=\frac{1}{N^2}\sum _{i\in V}\sum _{j\in V}\delta \left(s-c_{ij}^{*}\right),p_{Q^{*}}\left(s\right):=\frac{1}{T^2}\sum _{\alpha \in S}\sum _{\beta \in S}\delta (s-q_{*}^{\alpha \beta })$$

(61)

which in the following we call fluctuation distributions. Again, we can define analogues in the magnetic description. We can associate the spin–spin correlation matrix typical of systems seen in statistical mechanics

$$C:=\left\{c_{ij}\in \left[1-,1\right]:i,j\in V\right\},c_{ij}:=\frac{1}{T}\sum _{\alpha \in S}{\sigma }_i^{\alpha }{\sigma }_j^{\alpha }$$

(62)

and the overlap matrix commonly used in spin glass theory [56,60]

$$Q:=\{q^{\alpha \beta }\in \left[-1,1\right]:\alpha ,\beta \in S\},q^{\alpha \beta }:=\frac{1}{N}\sum _{i\in V}{\sigma }_i^{\alpha }{\sigma }_i^{\beta }$$

(63)

The relationship between kernels, correlations and overlap is still the same

$$M{M}^{†}/T=C,M^{†}M/N=Q$$

(64)

As before, we can define the mean field matrices

$$C_0:=\left\{m_i{m}_j\in \left[-1,1\right]:i,j\in V\right\},Q_0:=\{{\mu }^{\alpha }{\mu }^{\beta }\in \left[-1,1\right]:\alpha ,\beta \in S\}$$

(65)

however, it is important to note that the matrices constructed by the magnetization kernel have a different meaning than those constructed by the binary kernel. To compare them, one should again consider the connected matrices

$$C^{\ast }:=C-C_0,Q^{\ast }:=Q-Q_0$$

(66)

The latter contain only the correlations between fluctuations and are the same for both spin and binary neurons minus a scaling factor. Note that the distribution of the levels of $Q^{*}$ is exactly the distribution of the overlaps mentioned in the Replica Symmetry Breaking (RSB) theory [60].

3.2.5. Differences between Spin and Lattice Gas

Although from a mathematical point of view the two descriptions are equivalent, some observables may have different meanings. For example, in the case of activities, the correlation between spikes is one only when the neurons fire together and equal to zero in any other case, while the product between two spins is equal to one if the variables are equal and negative if the variables are opposite. In practice, the first variable is sensitive only to the event when the two neurons fire at the same time; the matrix of values is

$${\phi }_i^{\alpha }{\phi }_j^{\beta }=\left\{\begin{array}{cc}1& {\phi }_i^{\alpha }=1,{\phi }_j^{\beta }=1\\ 0& {\phi }_i^{\alpha }=1,{\phi }_j^{\beta }=0\\ 0& {\phi }_i^{\alpha }=0,{\phi }_j^{\beta }=1\\ 0& {\phi }_i^{\alpha }=0,{\phi }_j^{\beta }=0\end{array}\right.$$

(67)

putting on the same level (same null value) events in which only one neuron fires and when they are silent together. The second, on the other hand, distinguishes only whether or not they do the same thing, regardless of whether or not they fire

$${\sigma }_i^{\alpha }{\sigma }_j^{\beta }=\left\{\begin{array}{cc}1& {\sigma }_i^{\alpha }=+1,{\sigma }_j^{\beta }=+1\\ -1& {\sigma }_i^{\alpha }=+1,{\sigma }_j^{\beta }=-1\\ -1& {\sigma }_i^{\alpha }=-1,{\sigma }_j^{\beta }=+1\\ 1& {\sigma }_i^{\alpha }=-1,{\sigma }_j^{\beta }=-1\end{array}\right.$$

(68)

Notice that the relationship between the two observables is

$${\sigma }_i^{\alpha }{\sigma }_j^{\beta }=4{\phi }_i^{\alpha }{\phi }_j^{\beta }-2({\phi }_i^{\alpha }+{\phi }_j^{\beta })+1$$

(69)

and that this in turn identifies a new observable

$${\phi }_i^{\alpha }+{\phi }_j^{\beta }=\left\{\begin{array}{cc}2& {\phi }_i^{\alpha }=1,{\phi }_j^{\beta }=1\\ 1& {\phi }_i^{\alpha }=1,{\phi }_j^{\beta }=0\\ 1& {\phi }_i^{\alpha }=0,{\phi }_j^{\beta }=1\\ 0& {\phi }_i^{\alpha }=0,{\phi }_j^{\beta }=0\end{array}\right.$$

(70)

Linearly related to its corresponding one calculated with the magnetizations

$${\sigma }_i^{\alpha }+{\sigma }_j^{\beta }=2({\phi }_i^{\alpha }+{\phi }_j^{\beta })-1$$

(71)

explicitly the sum of the spins worth

$${\sigma }_i^{\alpha }+{\sigma }_j^{\beta }=\left\{\begin{array}{cc}2& {\sigma }_i^{\alpha }=+1,{\sigma }_j^{\beta }=+1\\ 0& {\sigma }_i^{\alpha }=+1,{\sigma }_j^{\beta }=-1\\ 0& {\sigma }_i^{\alpha }=-1,{\sigma }_j^{\beta }=+1\\ -2& {\sigma }_i^{\alpha }=-1,{\sigma }_j^{\beta }=-1\end{array}\right.$$

(72)

This variable allows us to distinguish three cases (both silent, both active only one active). Its intrinsic meaning is not yet clear, assuming it has any, however, if we take an exponential transformation of the variables

$$z_i^{\alpha }:=\exp \lambda {\sigma }_i^{\alpha },{\gamma }_i^{\alpha }:=\exp 2\lambda {\phi }_i^{\alpha }$$

(73)

we find that the equivalent relationship is with the products

$$z_i^{\alpha }{z}_j^{\beta }={\gamma }_i^{\alpha }{\gamma }_j^{\beta }\exp \left(-2\lambda \right).$$

(74)

Let us now consider the difference variable, which we will later interpret as a generalized form of impulse to construct the kinetic term of the Lagrangian

$${\phi }_i^{\alpha }-{\phi }_j^{\beta }=\left\{\begin{array}{cc}0& {\phi }_i^{\alpha }=1,{\phi }_j^{\beta }=1\\ 1& {\phi }_i^{\alpha }=1,{\phi }_j^{\beta }=0\\ -1& {\phi }_i^{\alpha }=0,{\phi }_j^{\beta }=1\\ 0& {\phi }_i^{\alpha }=0,{\phi }_j^{\beta }=0\end{array}\right.$$

(75)

the link with the spin counterpart is

$${\sigma }_i^{\alpha }-{\sigma }_j^{\beta }=2({\phi }_i^{\alpha }-{\phi }_j^{\beta }),$$

(76)

the difference between spin is the vector

$${\sigma }_i^{\alpha }-{\sigma }_j^{\beta }=\left\{\begin{array}{cc}0& {\sigma }_i^{\alpha }=+1,{\sigma }_j^{\beta }=+1\\ 2& {\sigma }_i^{\alpha }=+1,{\sigma }_j^{\beta }=-1\\ -2& {\sigma }_i^{\alpha }=-1,{\sigma }_j^{\beta }=+1\\ 0& {\sigma }_i^{\alpha }=-1,{\sigma }_j^{\beta }=-1\end{array}\right.$$

(77)

Notice that the value of the modulus for sums

$$|{\sigma }_i^{\alpha }+{\sigma }_j^{\beta }|=\left\{\begin{array}{cc}2& {\sigma }_i^{\alpha }=+1,{\sigma }_j^{\beta }=+1\\ 0& {\sigma }_i^{\alpha }=+1,{\sigma }_j^{\beta }=-1\\ 0& {\sigma }_i^{\alpha }=-1,{\sigma }_j^{\beta }=+1\\ 2& {\sigma }_i^{\alpha }=-1,{\sigma }_j^{\beta }=-1\end{array}\right.$$

(78)

is complementary to the modulus of differences

$$|{\sigma }_i^{\alpha }-{\sigma }_j^{\beta }|=\left\{\begin{array}{cc}0& {\sigma }_i^{\alpha }=+1,{\sigma }_j^{\beta }=+1\\ 2& {\sigma }_i^{\alpha }=+1,{\sigma }_j^{\beta }=-1\\ 2& {\sigma }_i^{\alpha }=-1,{\sigma }_j^{\beta }=+1\\ 0& {\sigma }_i^{\alpha }=-1,{\sigma }_j^{\beta }=-1\end{array}\right.$$

(79)

it follows that the moduli of sum and difference satisfy the relationship

$$|{\sigma }_i^{\alpha }+{\sigma }_j^{\beta }| + |{\sigma }_i^{\alpha }-{\sigma }_j^{\beta }| =2$$

(80)

and that the moduli of spin and neuron differences are directly proportional

$$|{\sigma }_i^{\alpha }-{\sigma }_j^{\beta }| =2|{\phi }_i^{\alpha }-{\phi }_j^{\beta }|$$

(81)

One can therefore write the modulus of the sum of spins with a quantity proportional to the modulus of the difference of neurons (or spins) changed by the sign. The formula for the product of the spins is therefore

$${\sigma }_i^{\alpha }{\sigma }_j^{\beta } = 1 - |{\sigma }_i^{\alpha }-{\sigma }_j^{\beta }| = 1-2|{\phi }_i^{\alpha }-{\phi }_j^{\beta }|$$

(82)

while that for the sum of the spin is

$${\sigma }_i^{\alpha }+{\sigma }_j^{\beta }=2{\sigma }_i^{\alpha }(1 - |{\sigma }_i^{\alpha }-{\sigma }_j^{\beta }\left|\right)=2(2{\phi }_i^{\alpha }-1)(1-2|{\phi }_i^{\alpha }-{\phi }_j^{\beta }\left|\right)$$

(83)

both proportional to the modulus of the difference between the neurons.

3.3. Estimators for Ergodicity Breaking

The idea of ergodicity is related to both space and time. With the term “ergodic”, we intend that the same information can be obtained by looking at a small portion of space for a large amount of time, or at a large portion of space for a small amount of time. Quoting Yamamoto’s book [130]: the idea of ergodicity arises if we have a single sample function of a stochastic process instead of the whole ensemble. A single sample function often provides little information about the statistics of the process. However, if the process is ergodic, that is, the time averages are equal to the ensemble averages, then all statistical information can be derived from a single sample function. When a process is ergodic, each sample function represents the entire process. Reflection should convince that the process must necessarily be stationary. Ergodicity thus implies stationarity. There are levels of ergodicity, just as there are levels (degrees) of stationarity. We will mostly consider two levels of ergodicity: ergodicity in mean and in correlation.

3.3.1. Ergodicity in Mean and Correlation

From Yamamoto’s book [130], a process is said to be “ergodic on average” if the averages of rows and columns of the kernel are equal

$${\mu }^{\alpha }=m_i$$

(84)

Note that this implies averages ${\mu }^{\alpha }$ and $m_i$ constants in $\alpha$ and i, which by the definitions given earlier should necessarily be equal to the offset $\overline{M}$. Let ${\chi }_i^k$ be the autocorrelation with period k of the i-th neuron

$${\chi }_i^k=\frac{1}{T}\sum _{\alpha \in S}{\sigma }_i^{\alpha }{\sigma }_i^{\alpha -k}$$

(85)

and let $q^{\alpha \alpha -k}$ be the overlap between the states at the time $\alpha$ and $\alpha -k$

$$q^{\alpha \alpha -k}=\frac{1}{N}\sum _{i\in V}{\sigma }_i^{\alpha }{\sigma }_i^{\alpha -k}$$

(86)

A process is said to be “ergodic in correlation” if

$${\chi }_i^k=q^{\alpha \alpha -k}$$

(87)

from which it follows that the autocorrelation and overlap functions must be constant in i and $\alpha$, respectively. Note that for a given k the averages are equal

$$\frac{1}{N}\sum _{i\in V}{\chi }_i^k=\frac{1}{T}\sum _{\alpha \in S}{q}^{\alpha \alpha -k}$$

(88)

and we can therefore introduce the period-averaged autocorrelation k

$${\mathsf{\Delta }}^k:=\frac{1}{T}\sum _{\alpha \in S}{q}^{\alpha \alpha -k}=\frac{1}{T}\sum _{\alpha \in S}\frac{1}{N}\sum _{i\in V}{\sigma }_i^{\alpha }{\sigma }_i^{\alpha -k}$$

(89)

which should highlight synchronous activity at a given time scale k (if any). In a correlation-ergodic process, this function describes the system completely. We should specify whether one should consider simple or connected correlations. If connected correlations are negligible, one has

$${\mathsf{\Delta }}_0^k:=\frac{1}{T}\sum _{\alpha \in S}{\mu }^{\alpha }{\mu }^{\alpha -k}$$

(90)

which can be deduced from the averaged kernel and should highlight the time scales at which first-order variables (i.e., averages) are correlated. Therefore, to study true second-order ergodicity, we need to look at the connected autocorrelation

$${\mathsf{\Delta }}_{*}^k:=\frac{1}{T}\sum _{\alpha \in S}\left(q^{\alpha \alpha -k}-{\mu }^{\alpha }{\mu }^{\alpha -k}\right)$$

(91)

which is also independent from the choice of representation for the computational cell (either spin or lattice gas). This quantity must necessarily be calculated by averaging the matrices and should track the correlation scales of the fluctuations in the considered time window. Note that for each of these observables it is also possible to calculate the error with a simple propagation: for a window of size T, the error diverges as k approaches T because of the reduction in the number of values over which the average is averaged.

3.3.2. Stationarity

Again, from Yamamoto’s book [130], a kernel is said to be stationary in a weak sense if the mean value of the columns ${\mu }^{\alpha }$ is a constant and the overlap $q^{\alpha \alpha -k}$ between the states at times $\alpha$ e $\alpha -k$ depends only on k (and not $\alpha$). If a process is stationary in the weak sense, the autocorrelation function and the power spectral density function form a pair of Fourier transforms (Wiener-Khinchine theorem). Therefore, if we know or can measure the autocorrelation function, we can find the power spectrum density function, that is, which frequencies contain how much power in the signal. Ergodicity in correlation implies stationarity, but stationarity does not imply ergodicity in correlation.

3.3.3. Ergodicity in Distribution

These notions of ergodicity can be relaxed, for example, by considering the distance between the distributions of means $p_m$ and $p_{\mu }$. This form of ergodicity is less restrictive than the previous ones since it does not require stationarity of the process; therefore, it is some weaker form of ergodicity than those described by Yamamoto and is probably non-standard, although it is quite natural to consider in the context of kernels [56].

3.3.4. Wasserstein Metric

An explicit definition of ergodicity in distributions requires defining a distance between distributions: a particularly interesting distance (essentially for its relevance in the context of optimal transport) is the Wasserstein metric of order k

$$W_k{\left(p_m,p_{\mu }\right)}^k:={\int }_0^{1}ds{\left|F_m^{-1}\left(s\right)-F_{\mu }^{-1}\left(s\right)\right|}^k$$

(92)

It can be shown that convergence with respect to a distance of order k is equivalent to the usual convergence in the weak sense plus the convergence of the first k moments. For time windows with $N=T$, the formula becomes particularly simple,

$$W_k{\left(p_m,p_{\mu }\right)}^k:=\frac{1}{N}\sum _{i\in V}{\left|m_{\left(i\right)}-{\mu }^{\left(i\right)}\right|}^k$$

(93)

for $N\ne T$ one must be careful to first establish an appropriate binning. The version that seems most interesting to us is that of order $k=1$, which is equivalent to weak convergence and is also known as the “earth mover” distance in that it establishes the optimal probability mass transport plan for transforming one distribution into the other. The distributions are thought of as two sand piles on the segment and the cost of moving one unit of mass from one value to the other is taken as proportional to the distance. In practice, it minimizes the work of transforming one pile into the other. We also have that the distance between cumulants is the same as with quantiles

$${\int }_0^{1}ds\left|F_m^{-1}\left(s\right)-F_{\mu }^{-1}\left(s\right)\right|={\int }_{-1}^{1}du\left|F_m\left(u\right)-F_{\mu }\left(u\right)\right|$$

(94)

However, it is important to remember that all these quantities are ideally designed to study limit kernels. For N and T finite (and not even particularly large in our case), the number of support levels accessible for the two distributions could be very different depending on how many clock times are in the window, given that the number of observed neurons remains fixed. One must therefore carefully consider whether there are enough events in the time interval under consideration such that the spectrum of m is reasonably comparable with that of $\mu$.

4. Lattice Field Theory

It has been proposed by many authors [41,115,131] that although QFTs are generally defined on a continuous support, it is perfectly possible to formulate physical theories directly in terms of difference equations and still keep all the desirable symmetries and conservation laws of continuous formulations. This is an example of a native lattice approach to quantum field theory and its practical importance is growing with the available computational power and evolution of AI. Let us introduce an index for the “mixed space” (or interval space)

$$\mathsf{\Lambda }:=\{1\le \mathcal{l}\le L\}$$

(95)

that is a vertex set of $L=NT$ vertices marked by the index ℓ, then let M be a spin field on $\mathsf{\Lambda }$ with components supported by $\left\{-1,1\right\}$. For now, we will represent the magnetization kernel as a spin vector on the mixed space–time lattice $\mathsf{\Lambda }$, which collects the value of the field at all intervals.

$$M:=\{{\sigma }_{\mathcal{l}}\in \mathsf{\Gamma }:\mathcal{l}\in \mathsf{\Lambda }\}$$

(96)

For the moment, M is simply a vector that contains the field value for all the space–time points, or computational cells in our case. We will soon re-map $\mathsf{\Lambda }$ into a multiplex lattice $VS$ where the proper time $\alpha$ (special dimension) is treated separately from the other dimensions (ordinary dimensions), that is, the “kernel representation” [56], but already at this point we can highlight another key idea of using a LFT to fit neural data. In contrast with what is usually done in max entropy approaches [62], in a LFT, the same neuron at two different times is considered in the same way as two different neurons, and it is the “action” function that ultimately correlates them in such a way that they look like the same neuron evolving in time. Then, let $\mathcal{O}$ be a test function of M. Following the stochastic quantization approach of Symanzik, Nelson, Parisi et al. [42,128,132,133,134], we postulate the analytic euclidean action function [41,43,73]

$$\mathcal{A}:{\mathbb{R}}^{\mathsf{\Lambda }}\to \mathbb{R}$$

(97)

and that the averages can be computed from the Wick-rotated Gell–Mann–Low (WGL) formula [133]. The combined work of several authors showed that this is equivalent to the Gibbs average [135] with $\mathcal{A}$ on behalf of the Hamiltonian, which corresponds to the principle of least action. The WGL formula is

$$\langle \mathcal{O}\left(X\right)\rangle =\sum _{X\in {\mathbb{R}}^{\mathsf{\Lambda }}}\mathcal{O}\left(X\right)\frac{\exp \left[-\lambda \mathcal{A}\left(X\right)\right]}{{\sum }_{X^{\prime }\in {\mathbb{R}}^{\mathsf{\Lambda }}}\exp \left[-\lambda \mathcal{A}\left(X^{\prime }\right)\right]}$$

(98)

and is suitable to describe any observable that depends on the field X. The continuous (or thermodynamic) limit of this theory is attained for $L\to \infty$, if it exists, while the zero temperature limit, $\lambda \to \infty$, correspond to the non-quantum limit of the theory.

4.1. Qubit Field Theory

Binary quantum field theories were pioneered by C. F. von Weizsacker in the 1950s with the “Ur” (Alternatives) theory [53,136]. The Ur theory is the earliest example of the Qubit field theory [53,54,55,56] and is probably the simplest of all lattice quantum field theories (QFTs). Here, we apply the Taylor theorem and other elementary mathematical methods to the Lee formulation of quantum mechanics [40,41] in order to obtain a path integral formulation of the Ur theory via perturbative methods. We start from the assumption that the action is an analytic function of the field components, then Taylor’s theorem can be applied to obtain a convergent perturbation theory [134]. Define the auxiliary functions

$$D_{\mathcal{l}}\left(X\right):=\frac{\partial \mathcal{A}\left(X\right)}{\partial x_{\mathcal{l}}},D_{\mathcal{l}{\mathcal{l}}^{\prime }}\left(X\right):=\frac{{\partial }^2\mathcal{A}\left(X\right)}{\partial x_{\mathcal{l}}\partial x_{{\mathcal{l}}^{\prime }}}\left[1-\mathbb{I}(\mathcal{l}={\mathcal{l}}^{\prime })\right]+\frac{1}{2}\frac{{\partial }^2\mathcal{A}\left(X\right)}{\partial x_{\mathcal{l}}^2}\mathbb{I}(\mathcal{l}={\mathcal{l}}^{\prime })$$

(99)

then, by Taylor’s theorem, the action can be expanded around the null field and this correspond to the one- and two- vertex interactions, etc.

$$\mathcal{A}\left(X\right)=\sum _{\mathcal{l}\in \mathsf{\Lambda }}{D}_{\mathcal{l}}\left(0\right)x_{\mathcal{l}}+\sum _{\mathcal{l}\in \mathsf{\Lambda }}\sum _{{\mathcal{l}}^{\prime }\in \mathsf{\Lambda }}{D}_{\mathcal{l}{\mathcal{l}}^{\prime }}\left(0\right)x_{\mathcal{l}}{x}_{{\mathcal{l}}^{\prime }}+\cdots$$

(100)

here, we stop at the second order to avoid complications, but a fourth-order theory should be considered for an accurate description of physical theories. To obtain a Ur theory we can take $|x_{\mathcal{l}}|\to g$ with $g<1$ (a form of Ising limit [137]). Let us introduce the tensors

$$F_{\mathcal{l}}:=g{D}_{\mathcal{l}}\left(0\right),F_{\mathcal{l}{\mathcal{l}}^{\prime }}:=g^2{D}_{\mathcal{l}{\mathcal{l}}^{\prime }}\left(0\right),\cdots$$

(101)

By substituting into the series expansion before we obtain the first-order perturbation theory of the Ur in magnetic representation

$$\mathcal{A}\left(M\right)=\sum _{\mathcal{l}\in \mathsf{\Lambda }}{F}_{\mathcal{l}}{\sigma }_{\mathcal{l}}+\sum _{\mathcal{l}\in \mathsf{\Lambda }}\sum _{{\mathcal{l}}^{\prime }\in \mathsf{\Lambda }}{F}_{\mathcal{l}{\mathcal{l}}^{\prime }}{\sigma }_{\mathcal{l}}{\sigma }_{{\mathcal{l}}^{\prime }}+\cdots$$

(102)

the theory is controlled by the tensor sequence F. We can immediately recognize the Ising-like structure of the action, which can be related to the usual formulations of the Standard Model on the lattice trough, for example, the Parotto mapping of QCD [46]. In general, the statistical method [42,56,60] allows the problem of finding the quantum (thus, also classical) time evolution of a system of interacting binary fields [53,54,55,56] to be transformed into a problem of classical statistical mechanics on a lattice [41,42,43,44], which can then be studied through canonical theory [74,135], renormalization [38,101,102,103,104,105] and other powerful mathematical methods [56,60].

4.1.1. Neural LFT

In the following section we provide the complete derivation of the expressions of the action $\mathcal{A}$, both in the binary case and in the spin representation. Therefore, for the sake of completeness, some expressions and definitions given in the main text will be included again. For any physical (then limited) region of our analogue space–time, a map $\mathsf{\Theta }$ exists that connects the kernel with the mixed space and vice versa. Let us introduce a “grand map”

$$\mathsf{\Theta }:VS\to \mathsf{\Lambda },{\mathsf{\Theta }}^{-1}:\mathsf{\Lambda }\to VS$$

(103)

that establishes a biunivocal relation between the points of the mixed space $\mathsf{\Lambda }$ and those of the observed space time VS. This map always exists for any physical (finite) discrete observable and is another free parameter of the theory that can be tuned to highlight the space–time structures. Then, we relabel the points according to a double index as in [56]. Assuming that the neuron’s computation is supported by the the field

$${\psi }_i^{\alpha }:={\phi }_i^{\alpha }-\overline{\mathsf{\Omega }},$$

(104)

where $\overline{\mathsf{\Omega }}$ is the global offset for the neuron in vivo, which could possibly be zero. By Taylor’s theorem, the action of a lattice field theory $\mathcal{A}\left(\mathsf{\Omega }\right|F)$ can be described by an expansion of the kind:

$$\begin{array}{ll}\mathcal{A}\left(\mathsf{\Omega }\right|F)& ={\displaystyle \sum _{i\in V}}{\displaystyle \sum _{\alpha \in S}}{F}_i^{\alpha }{\psi }_i^{\alpha }+{\displaystyle \sum _{i\in V}}{\displaystyle \sum _{j\in V}}{\displaystyle \sum _{\alpha \in S}}{\displaystyle \sum _{\beta \in S}}{F}_{ij}^{\alpha \beta }{\psi }_i^{\alpha }{\psi }_j^{\beta }+\\ & +{\displaystyle \sum _{i\in V}}{\displaystyle \sum _{j\in V}}{\displaystyle \sum _{h\in V}}{\displaystyle \sum _{\alpha \in S}}{\displaystyle \sum _{\beta \in S}}{\displaystyle \sum _{\gamma \in S}}{F}_{ijh}^{\alpha \beta \gamma }{\psi }_i^{\alpha }{\psi }_j^{\beta }{\psi }_h^{\gamma }+{\displaystyle \sum _{i\in V}}{\displaystyle \sum _{j\in V}}{\displaystyle \sum _{h\in V}}{\displaystyle \sum _{k\in V}}{\displaystyle \sum _{\alpha \in S}}{\displaystyle \sum _{\beta \in S}}{\displaystyle \sum _{\gamma \in S}}{\displaystyle \sum _{\delta \in S}}{F}_{ijhk}^{\alpha \beta \gamma \delta }{\psi }_i^{\alpha }{\psi }_j^{\beta }{\psi }_h^{\gamma }{\psi }_k^{\delta }+\cdots \end{array}$$

(105)

Each term represent one-, two-, three- and four-vertex interactions, etc., while the tensors sequence F collects the parameters of the theory. However, if we want to consider the same non-relativistic approximation used by Schneidman and colleagues [62], interactions with more than two vertices can be neglected as well as two-vertex interactions with four different indices. Therefore, the proposed action reduces to:

$$\mathcal{A}\left(\mathsf{\Omega }\right|A,B)=\sum _{i\in V}\sum _{j\in V}{A}_{ij}\sum _{\alpha \in S}{\psi }_i^{\alpha }{\psi }_j^{\alpha }+\sum _{\alpha \in S}\sum _{\beta \in S}{B}^{\alpha \beta }\sum _{i\in V}{\psi }_i^{\alpha }{\psi }_i^{\beta }$$

(106)

The action depends explicitly on the correlation and overlap matrices and is controlled by the matrix of potential interactions A and by the matrix of kinetic interactions B. We can switch to the binary representation $\phi$ through the transformation

$${\psi }_i^{\alpha }{\psi }_j^{\beta }={\phi }_i^{\alpha }{\phi }_j^{\beta }-\overline{\mathsf{\Omega }}({\phi }_j^{\beta }+{\phi }_i^{\alpha })+{\overline{\mathsf{\Omega }}}^2$$

(107)

doing the algebra one finds that the structure of the theory is the same:

$$\begin{array}{c}\sum _{i\in V}\sum _{j\in V}{A}_{ij}\sum _{\alpha \in S}{\psi }_i^{\alpha }{\psi }_i^{\beta }+\sum _{\alpha \in S}\sum _{\beta \in S}{B}^{\alpha \beta }\sum _{i\in V}{\psi }_i^{\alpha }{\psi }_i^{\beta }=\hfill \\ =\sum _{i\in V}\sum _{j\in V}{A}_{ij}\sum _{\alpha \in S}{\phi }_i^{\alpha }{\phi }_j^{\alpha }+\sum _{\alpha \in S}\sum _{\beta \in S}{B}^{\alpha \beta }\sum _{i\in V}{\phi }_i^{\alpha }{\phi }_i^{\beta }+\\ \hfill +\overline{\mathsf{\Omega }}\sum _{\alpha \in S}\sum _{i\in V}{\phi }_i^{\alpha }\sum _{j\in V}(A_{ij}-A_{ji})+\overline{\mathsf{\Omega }}\sum _{\alpha \in S}\sum _{i\in V}{\phi }_i^{\alpha }\sum _{\beta \in S}(B^{\alpha \beta }-B^{\beta \alpha })+\mathrm{const}.\end{array}$$

(108)

because the connected part is identical. If the global offset $\overline{\mathsf{\Omega }}$ is not zero one must take into account the appearance of additional currents

$$I_i^0:=\overline{\mathsf{\Omega }}\sum _{j\in V}(A_{ij}-A_{ji}),I_0^{\alpha }:=\overline{\mathsf{\Omega }}\sum _{\beta \in S}(B^{\alpha \beta }-B^{\beta \alpha })$$

(109)

which, however, transform linearly. Ultimately, the action in the binary form is:

$$\mathcal{A}\left(\mathsf{\Omega }|A,B\right)=\sum _{i\in V}{I}_i^0\sum _{\alpha \in S}{\phi }_i^{\alpha }+\sum _{\alpha \in S}{I}_0^{\alpha }\sum _{i\in V}{\phi }_i^{\alpha }+\sum _{i\in V}\sum _{j\in V}{A}_{ij}\sum _{\alpha \in S}{\phi }_i^{\alpha }{\phi }_j^{\alpha }+\sum _{\alpha \in S}\sum _{\beta \in S}{B}^{\alpha \beta }\sum _{i\in V}{\phi }_i^{\alpha }{\phi }_i^{\beta }$$

(110)

Notice that by normalizing the sums the action can be rewritten using the first-order observables f, $\omega$, which are obtained from the kernel $\mathsf{\Omega }$ through linear transformations, and with those of the second order, the matrices $\mathsf{\Phi }$ and $\mathsf{\Pi }$:

$$\mathcal{A}\left(\mathsf{\Omega }|A,B\right)=T\sum _{i\in V}{I}_i^0{f}_i+N\sum _{\alpha \in S}{I}_0^{\alpha }{\omega }^{\alpha }+T\sum _{i\in V}\sum _{j\in V}{A}_{ij}{\varphi }_{ij}+N\sum _{\alpha \in S}\sum _{\beta \in S}{B}^{\alpha \beta }{p}^{\alpha \beta },$$

(111)

that is Equation (7) of the main text. For a single realization of the process the kernel is binary and the matrices can be obtained from the kernel (in this case the hypermatrix is a redundant representation). However, as we shall see, this does not apply in general to the averaged hypermatrix, where the correlations contain information about the ensemble fluctuations.

4.1.2. Magnetic Representation

To switch to the magnetic representation we apply the usual spin-bit transformation $\sigma =2\phi -1$:

$$\begin{array}{c}\mathcal{A}\left(M|A,B\right)=\frac{1}{2}\sum _{i\in V}{I}_i^0\sum _{\alpha \in S}{\sigma }_i^{\alpha }+\frac{1}{2}\sum _{\alpha \in S}{I}_0^{\alpha }\sum _{i\in V}{\sigma }_i^{\alpha }+\hfill \\ +\frac{1}{4}\sum _{i\in V}\sum _{j\in V}(A_{ij}+A_{ij})\sum _{\alpha \in S}{\sigma }_i^{\alpha }+\frac{1}{4}\sum _{\alpha \in S}\sum _{\beta \in S}(B^{\alpha \beta }+B^{\beta \alpha })\sum _{i\in V}{\sigma }_i^{\alpha }+\\ \hfill +\frac{1}{4}\sum _{i\in V}\sum _{j\in V}{A}_{ij}\sum _{\alpha \in S}{\sigma }_i^{\alpha }{\sigma }_j^{\alpha }+\frac{1}{4}\sum _{\alpha \in S}\sum _{\beta \in S}{B}^{\alpha \beta }\sum _{i\in V}{\sigma }_i^{\alpha }{\sigma }_i^{\beta }\end{array}$$

(112)

The structure of the interaction is identical except for a global rescaling

$${\tilde{A}}_{ij}=\frac{1}{4}{A}_{ij},{\tilde{B}}^{\alpha \beta }:=\frac{1}{4}{B}^{\alpha \beta },$$

(113)

and an adjustment of currents with an additional term

$${\tilde{I}}_i^0:=2\overline{\mathsf{\Omega }}\sum _{j\in V}({\tilde{A}}_{ij}-{\tilde{A}}_{ij})+\sum _{j\in V}({\tilde{A}}_{ij}+{\tilde{A}}_{ji}),{\tilde{I}}_0^{\alpha }:=2\overline{\mathsf{\Omega }}\sum _{\beta \in S}({\tilde{B}}^{\alpha \beta }-{\tilde{B}}^{\beta \alpha })+\sum _{\beta \in S}({\tilde{B}}^{\alpha \beta }+{\tilde{B}}^{\beta \alpha }).$$

(114)

The action in the magnetic representation is therefore

$$\mathcal{A}\left(M\right|\tilde{A},\tilde{B})=T\sum _{i\in V}{\tilde{I}}_i^0{m}_i+N\sum _{\alpha \in S}{\tilde{I}}_0^{\alpha }{\mu }^{\alpha }+T\sum _{i\in V}\sum _{j\in V}{\tilde{A}}_{ij}{c}_{ij}+N\sum _{\alpha \in S}\sum _{\beta \in S}{\tilde{B}}^{\alpha \beta }{q}^{\alpha \beta }$$

(115)

In this case the hypermatrix will consists of M, C and Q. We can recover the Ising Hamiltonian used in Schneidman et al. [62] in the limit $T\to 1$ and $B\to 0$:

$$\mathcal{A}\left(M\right|\tilde{A},0)=\sum _{i\in V}\sum _{j\in V}{\tilde{A}}_{ij}{\sigma }_i^1{\sigma }_j^1$$

(116)

Thus, the max entropy principle is recovered as specific case of a field theory with zero kinetic energy. Since the theories are equivalent in the coming manipulations we will mainly use the spin representation.

4.1.3. Lagrangian Description of LFT

Assuming that the process evolves causally, it follows that the kinetic matrix B must be upper triangular, that is, the state of the system at instant $\alpha$ depends only on the states realized in the previous $\beta \le \alpha -1$. Therefore, we can define the sequence of time windows

$$\mathcal{S}:=\left\{S_{\alpha }\subset S:\alpha \in S\right\},S_{\alpha }:=\left\{1\le \beta \le \alpha \right\}$$

(117)

In this way it is possible to define the Lagrangian of the system

$$\begin{array}{c}\mathcal{L}\left({\sigma }_V^{S_{\alpha }}\right|A,B):=-\sum _{i\in V}\sum _{j\in V}{A}_{ij}{\sigma }_i^{\alpha }{\sigma }_j^{\alpha }-\sum _{\beta \in S_{\alpha -1}}{B}^{\alpha \beta }\sum _{i\in V}{\sigma }_i^{\alpha }{\sigma }_i^{\beta }=\\ \hfill =-\sum _{i\in V}\sum _{j\in V}{A}_{ij}{\sigma }_i^{\alpha }{\sigma }_j^{\alpha }-N\sum _{\beta \in S_{\alpha -1}}{B}^{\alpha \beta }{q}^{\alpha \beta }\end{array}$$

(118)

where in the second line the definition of overlap $q^{\alpha \beta }$ is applied

$$\mathcal{A}\left(M|A,B\right)=\sum _{\alpha \in S}\mathcal{L}\left({\sigma }_V^{S_{\alpha }}\right|A,B).$$

(119)

We can isolate the potential term from the kinetic term (which depends on the overlap)

$$H\left({\sigma }_V^{\alpha }\right|A):=\sum _{i\in V}\sum _{j\in V}{A}_{ij}{\sigma }_i^{\alpha }{\sigma }_j^{\alpha },K\left(q^{\alpha S_{\alpha -1}}\right|B):=-N\sum _{\beta \in S_{\alpha -1}}{B}^{\alpha \beta }{q}^{\alpha \beta }.$$

(120)

Thus, we can rewrite the Lagrangian in the canonical form [120]

$$\mathcal{L}\left({\sigma }_V^{S_{\alpha }}\right|A,B)=-H\left({\sigma }_V^{\alpha }\right|A)+K\left(q^{\alpha S_{\alpha -1}}\right|B),$$

(121)

where $q^{\alpha S_{\alpha -1}}$ is the $\alpha$-th row of the matrix of overlaps up to the time $\alpha -1$

$$q^{\alpha S_{\alpha -1}}:=\{q^{\alpha \beta }\in Q:\beta \in S_{\alpha -1}\}$$

(122)

and this is enough to set the dynamics of the system. That the overlap-dependent term can be truly interpreted as a kinetic term is deduced by comparing with a simple Lagrangian system (see the work from Lee [41] for an overview). We introduce the pulse (or “momentum”) kernel

$$\partial M:=\left\{\partial {\sigma }_i^{\alpha }\in \left\{-2,0,2\right\}:\alpha \in S_{\alpha }\right\},\partial {\sigma }_i^{\alpha }:={\sigma }_i^{\alpha }-{\sigma }_i^{\alpha -1}$$

(123)

The Lagrangian of the scalar field is

$$\mathcal{L}({\sigma }_V^{\alpha },\partial {\sigma }_V^{\alpha }|A,B_0):=-H\left({\sigma }_V^{\alpha }\right|A)+\frac{1}{2}{B}_0{∥\partial {\sigma }_V^{\alpha }∥}_2^2$$

(124)

with a few algebraic steps (e.g., Babylonian trick [138]) it can be shown that

$${∥\partial {\sigma }_V^{\alpha }∥}_2^2=2N(1-q^{\alpha \alpha -1})$$

(125)

therefore, the Lagrangian can be rewritten as

$$\mathcal{L}({\sigma }_V^{\alpha },\partial {\sigma }_V^{\alpha }|A,B_0)=-H\left({\sigma }_V^{\alpha }\right|A)+B_{0}N(1-q^{\alpha \alpha -1}).$$

(126)

In the case of our action, taking

$$B^{\alpha \beta }=-B_0\mathbb{I}\left(\alpha -1=\beta \right)$$

(127)

where $\mathbb{I}(·)$ is the indicator function. The associated Lagrangian becomes

$$\mathcal{L}\left({\sigma }_V^{S_{\alpha }}\right|A,B)=-H\left({\sigma }_V^{\alpha }\right|A)-B_{0}N{q}^{\alpha \alpha -1}$$

(128)

and one can see immediately that the difference between the two Lagrangians is

$$\mathcal{L}\left({\sigma }_V^{S_{\alpha }}\right|A,B)-\mathcal{L}({\sigma }_V^{\alpha },\partial {\sigma }_V^{\alpha }|A,B_0)=-B_{0}N$$

(129)

i.e., a constant that is irrelevant to the determination of dynamics. Moreover, the sign of the kinetic term is reversed with respect to that of the overlap term. It follows that the free field system is a sub-case of the general action described at the beginning, whose overlap term can be reduced to the kinetic term of the free Lagrangian.

4.2. Statistical Field Theory

So far, our theory is equivalent to a binary quantum field theory on the lattice, i.e., the Qubit field theory [53,54,55,56], since the same results can be deduced by applying the Wick rotation, (i.e., a rotation $i\to -1$ of the imaginary time units into the real plane) [47,139,140] to a system of non-relativistic quantum oscillators [21,23,38,40,41,42,43,44,46,49,51,141]. In general, the evolution of a Lagrangian system is determined by the principle of stationary action, which means that the kernel that satisfies it is not necessarily a minimum of the action: it can also be a maximum, or a saddle point. Following Symanzik, Nelson, Parisi et al. [42,128,133,134], for the quantum evolution we consider a Gibbs principle [74] applied to the action, which is equivalent to the principle of least action [47,73]. We define the action’s partition function:

$$G\left(A,B\right)=\sum _{M\in {\left\{-1,1\right\}}^{VS}}\exp \left[-\lambda \mathcal{A}\left(M|A,B\right)\right]$$

(130)

where we interpret the action as a Hamiltonian and look for its minimum. Here, $\lambda$ is the inverse Planck constant and plays the role of a temperature. The classical limit is recovered for $\lambda \to \infty$. We also define the free action

$$\mathsf{\Psi }\left(A,B\right):=-\frac{1}{\lambda }logG\left(A,B\right)$$

(131)

which would be the analogue of the free energy. We then apply the steps to obtain the Gibbs principle [56]: first we manipulate the partition function, multiplying and dividing by a test measure to obtain the flat functional

$$\sum _{M\in {\left\{-1,1\right\}}^{VS}}\exp \left[-\lambda \mathcal{A}\left(M|A,B\right)\right]={\langle \exp \left[-\lambda \mathcal{A}\left(M|A,B\right)-log\zeta \left(M\right)\right]\rangle }_{\zeta }.$$

(132)

Then, we apply Jensen’s inequality to the average versus the test measure

$$\begin{array}{c}{\langle \exp \left[-\lambda \mathcal{A}\left(M|A,B\right)-log\zeta \left(M\right)\right]\rangle }_{\zeta }\ge \hfill \\ \ge \exp \left[-\lambda {\langle \mathcal{A}\left(M|A,B\right)\rangle }_{\zeta }-{\langle log\zeta \left(M\right)\rangle }_{\zeta }\right]=\exp \left[-\lambda \mathcal{F}\left(\zeta |A,B\right)\right]\hfill \end{array}$$

(133)

so as to obtain the free action functional

$$\mathcal{F}\left(\zeta |A,B\right):={\langle \mathcal{A}\left(M|A,B\right)\rangle }_{\zeta }+\frac{1}{\lambda }{\langle log\zeta \left(\sigma \right)\rangle }_{\zeta }$$

(134)

This functional is greater or equal to the free action for any test measure

$$\mathsf{\Psi }\left(A,B\right)\le \mathcal{F}\left(\zeta |A,B\right),\forall \zeta \in \mathcal{P}\left({\left\{-1,1\right\}}^{VS}\right)$$

(135)

and one can see that the minimum is actually reached by the Gibbs measure

$$\eta \left(M|A,B\right):=\frac{1}{G\left(A,B\right)}\exp \left[-\lambda \mathcal{A}\left(M|A,B\right)\right]$$

(136)

It can be verified that the measure satisfies the relationship with the free action

$$\mathcal{F}\left(\eta |A,B\right):=\underset{\zeta \in \mathcal{P}\left({\left\{-1,1\right\}}^{VS}\right)}{inf}\mathcal{F}\left(\zeta |A,B\right)=\mathsf{\Psi }\left(A,B\right)$$

(137)

If the system is assumed to be classical (i.e., non-quantum) the dynamic is obtained in the zero temperature limit. However, it could also have an intrinsic minimum temperature (equivalent to the non-zero Planck’s constant).

4.2.1. Connection with Replica Theory

The theory in the Lagrangian form allows us to establish a connection with the replica theory [60,87]

$$G\left(A,B\right)=\sum _{M\in {\left\{-1,1\right\}}^{VS}}\prod _{\alpha \in S}\exp [-\lambda \mathcal{L}\left({\sigma }_V^{S_{\alpha }}\right|A,B)]$$

(138)

and since the sum over the kernels is equivalent to a sum over T replicas of the system ${\sigma }_V$

$$\sum _{M\in {\left\{-1,1\right\}}^{VS}}=\sum _{{\sigma }_V^1\in {\left\{-1,1\right\}}^V}\sum _{{\sigma }_V^2\in {\left\{-1,1\right\}}^V}\cdots \sum _{{\sigma }_V^T\in {\left\{-1,1\right\}}^V}$$

(139)

the replicated system is obtained in the limit $B\to 0$ (no kinetic term)

$$\mathcal{A}\left(M|A,0\right)=-\sum _{\alpha \in S}H\left({\sigma }_V^{\alpha }\right|A)$$

(140)

with simple steps we arrive at the following relations

$$\begin{array}{c}G\left(A,0\right):=\sum _{\sigma \in {\left\{-1,1\right\}}^{VS}}\prod _{\alpha \in S}\exp [-\lambda H\left({\sigma }_V^{\alpha }\right|A)]=\hfill \\ =\sum _{{\sigma }_V^1\in {\left\{-1,1\right\}}^V}\exp [-\lambda H\left({\sigma }_V^1\right|A)]\cdots \sum _{{\sigma }_V^T\in {\left\{-1,1\right\}}^V}\exp [-\lambda H\left({\sigma }_V^T\right|A)]=\prod _{\alpha \in S}Z\left(A\right)=Z{\left(A\right)}^T\hfill \end{array}$$

(141)

where Z is the partition function associated with the Hamiltonian H and

$$Z\left(A\right):=\sum _{{\sigma }_V^1\in {\left\{-1,1\right\}}^V}\exp \left[-\lambda H\left({\sigma }_V^1\right|A)\right]$$

(142)

As one can see, the partition function of the action converges to the partition function of the Hamiltonian replicated T times. The interpretation of the replica trick [60]

$$logZ\left(A\right)=\underset{T\to 0}{lim}\frac{1}{T}[Z{\left(A\right)}^T-1]=\underset{T\to 0}{lim}\frac{1}{T}\left[G\left(A,0\right)-1\right]$$

(143)

is natural enough in this context: the formal limit $T\to 0$ describes a situation in which the continuous limit of the theory ($\tau \to 0$) is observed for an infinitesimal time.

4.2.2. External Input

So far, we have only considered the evolution of an isolated system, but obviously in our case the input is crucial, so we must include it in the model. This can be done in a relatively simple way by introducing the input kernel, which describes the input signal in the network

$$\mathcal{I}\left(M|I\right):=-\sum _{\alpha \in S}{I}_V^{\alpha }·{\sigma }_V^{\alpha }$$

(144)

which should be added to the action to obtain the description of the full system

$$\mathcal{A}\left(M|A,B,I\right):=\mathcal{A}\left(M|A,B\right)-\mathcal{I}\left(M|I\right)$$

(145)

By introducing the input partition function (see the Interface Model of [56])

$$R\left(I\right):=\sum _{M\in {\left\{-1,1\right\}}^{VS}}\exp \left[-\lambda \mathcal{I}\left(M|I\right)\right]=\prod _{\alpha \in S}\prod _{i\in V}2cosh\left(\lambda I_i^{\alpha }\right)$$

(146)

and applying the Gibbs principle we find the distribution of the input

$$\rho \left(M|I\right):=\frac{1}{R\left(I\right)}\exp \left[-\lambda \mathcal{I}\left(M|I\right)\right]=\frac{1}{R\left(I\right)}\prod _{\alpha \in S}\prod _{i\in V}\exp \left(-\lambda I_i^{\alpha }{\sigma }_i^{\alpha }\right)$$

(147)

The partition function of the general action can be expressed in terms of the average of the isolated state with respect to $\rho$

$$G\left(A,B,I\right)=R\left(I\right){\langle \exp \left[-\lambda \mathcal{A}\left(M|A,B\right)\right]\rangle }_{\rho }$$

(148)

Note that the partition can also be expressed as the average of the input over the measure of the isolated system

$$G\left(A,B,I\right)=G\left(A,B\right){\langle \exp \left[-\lambda \mathcal{I}\left(M|I\right)\right]\rangle }_{\eta }$$

(149)

from which a relationship between partition functions and averages over states

$$R\left(I\right){\langle \exp \left[-\lambda \mathcal{A}\left(M|A,B\right)\right]\rangle }_{\rho }=G\left(A,B\right){\langle \exp \left[-\lambda \mathcal{I}\left(M|I\right)\right]\rangle }_{\eta }$$

(150)

For example, the input kernel could model the signal arriving to the observed cortical area after a stimulus. In case of a motor task [35,86], which could be the thalamic input arriving to the boundary neurons (in a topological sense) of the recorded cortical region following the Go stimulus, it is expected to be a steady-state almost everywhere except around the time interval at which the motor plan is realized. If axonal and synaptic connections are reasonably stable then most of the observed variability could come from the input noise from the rest of the network, or slightly different initial conditions, etc. To include all possible effects one can introduce a “quenched” space–time noise term, i.e., a random field $\delta$ to be added to the input term I

$$\mathcal{I}\left(M|I,\delta \right):=-\sum _{\alpha \in S}{I}_V^{\alpha }·{\sigma }_V^{\alpha }-\sum _{\alpha \in S}{\delta }_V^{\alpha }·{\sigma }_V^{\alpha }$$

(151)

which statistically mimic the input noise on the time scale of the entire session. In case of the recordings described in the main text [35,86], we expect that quenched noise terms can be ignored.

4.2.3. Ground State of the Action and Order Parameter

The variational principle identifies a distribution $\eta$ called the ground state of the action (GS), which would be the one from which the mutielectrode interface draws the states we observe at the single trial level. Note that the GS of the action is a distribution in the space of kernels ${\{-1,1\}}^{VS}$, and hence a natural order parameter would be the kernel of the GS

$${\langle M\rangle }_{\eta }:=\left\{{\langle {\sigma }_i^{\alpha }\rangle }_{\eta }\in \mathbb{R}:i\in V,\alpha \in S\right\}.$$

(152)

The kernel of the GS is of particular interest since one can directly obtain the average momentum kernel ${\langle \partial M\rangle }_{\eta }$ and all kernels derived from linear operations. It also allows us to determine and subtract the steady state of the “hold” phase before the motor plan is observed (see main text). Notice that averages of first-order observables, such as the offset, or row and column averages, can be computed directly from the average kernel because they are related to it by a linear relationship. Additionally, if the connected correlation matrices are negligible then the correlation matrices can be deduced from the average kernel since $C_0$ and $Q_0$ depend on the averages. The amplitude of the kernel cell fluctuations satisfies the relation

$${\langle {[{\sigma }_i^{\alpha }-{\langle {\sigma }_i^{\alpha }\rangle }_{\eta }]}^2\rangle }_{\eta }=1-{\langle {\sigma }_i^{\alpha }\rangle }_{\eta }^2$$

(153)

However, the relationship does not apply in general,

$$\begin{array}{c}{\langle [{\sigma }_i^{\alpha }-{\langle {\sigma }_i^{\alpha }\rangle }_{\eta }][{\sigma }_j^{\beta }-{\langle {\sigma }_j^{\beta }\rangle }_{\eta }]\rangle }_{\eta }=\hfill \\ ={\langle {\sigma }_i^{\alpha }{\sigma }_j^{\beta }\rangle }_{\eta }-{\langle {\sigma }_i^{\alpha }\rangle }_{\eta }{\langle {\sigma }_j^{\beta }\rangle }_{\eta }-{\langle {\sigma }_i^{\alpha }\rangle }_{\eta }{\langle {\sigma }_j^{\beta }\rangle }_{\eta }+{\langle {\sigma }_i^{\alpha }\rangle }_{\eta }{\langle {\sigma }_j^{\beta }\rangle }_{\eta }={\langle {\sigma }_i^{\alpha }{\sigma }_j^{\beta }\rangle }_{\eta }-{\langle {\sigma }_i^{\alpha }\rangle }_{\eta }{\langle {\sigma }_j^{\beta }\rangle }_{\eta }\hfill \end{array}$$

(154)

Thus, the average kernel may not be a sufficient order parameter to fully describe the GS $\eta$, and we should also look at second-order variables. To verify this we can compute the ensemble covariance matrices

$${\langle \delta C\rangle }_{\eta }:={\langle C\rangle }_{\eta }-{\langle M\rangle }_{\eta }{\langle M\rangle }_{\eta }^{†}/T,{\langle \delta Q\rangle }_{\eta }:={\langle Q\rangle }_{\eta }-{\langle M\rangle }_{\eta }^{†}{\langle M\rangle }_{\eta }/N.$$

(155)

If these matrices are non-trivial it means that the correlations cannot be reconstructed from the average kernel. Thus, the most general order parameter in this approximation (two-body non-relativistic) is the hypermatrix, composed by the average kernel ${\langle M\rangle }_{\eta }$ and the average correlation matrices ${\langle Q\rangle }_{\eta }$ and ${\langle C\rangle }_{\eta }$.

4.2.4. Repeated Experiments and Ensemble Average

Let us see how we might model an actual experimental observation of these matrices. A possible method is to replicate an experiment n times and find a way to average the results in such a way as to estimate the ensemble average. First, we need to index the trials with the label k and the span of the index is denoted by

$$W:=\left\{1\le k\le n\right\},$$

(156)

the empirical ensemble collecting the actually observed kernels, called a recording session in [35], can be represented as follows

$$\mathcal{W}:=\{M_k\in {\left\{-1,1\right\}}^{VS}:k\in W\},$$

(157)

where $M_k$ is the k-th trial of the session

$$M_k:=\{{\phi }_{ik}^{\alpha }\in \left\{0,1\right\}:i\in V,\alpha \in S\}.$$

(158)

We added the new trial index k in the lower part of the $\phi$ symbol, though in principle it should have been placed on top as the idea of repeated experiments would suggest a kind of time variable. Since the trials are usually designed to be independent, the order in which they are performed should not matter for the k index. This is an ideal situation that should be verified for each experimental set, but it is also a reasonable approximation of the experimenter’s intentions. To confront the experiments and precisely define the empirical averages, we need one last ingredient: to choose the proper synchronization of the experimental kernels. Then, we introduce the integer vector

$${\nu }_W:=\left\{{\nu }_W\in \mathbb{Z}:k\in W\right\}$$

(159)

that collects the relative time shifts of the trials, i.e., for the k-th trial the $\alpha$ index is shifted by ${\nu }_k$ units of clock time, that is equivalent to apply the substitution

$$\alpha \to \alpha +{\nu }_k$$

(160)

We formally indicate the application of the timeshifts to the empirical ensemble with

$$\mathcal{W}\to \mathcal{W}\left({\nu }_W\right).$$

(161)

We argue that for some ${\nu }_W$, the average on the empirical set converges to the ensemble average in the ideal limit of infinite repetitions of the same experiments

$${\langle \mathcal{O}\left(M\right)\rangle }_{\eta }=\underset{n\to \infty }{lim}\frac{1}{n}\sum _{M\in \mathcal{W}\left({\nu }_W\right)}\mathcal{O}\left(M\right).$$

(162)

For example, the hypermatrix shown in Figure 1 is obtained by choosing ${\nu }_W$ in such a way as to synchronize the replicas of the experiment with respect to the movement onset (see Section 5.2.2), while in [35], the alignment is by Go signal. In principle, one should also consider more complex kinds of synchronizations, like alignments that are based on maximizing the correlations between the trials. Let us introduce the overlap matrix for the session

$$\mathcal{Q}\left({\nu }_W\right):=\{{\mathcal{Q}}_{k{k}^{\prime }}\left({\nu }_W\right)\in \left[0,1\right]:k,k^{\prime }\in W\},$$

(163)

where the entries are the overlaps between the trials k and $k^{\prime }$ with timeshifts ${\nu }_W$,

$${\mathcal{Q}}_{k{k}^{\prime }}\left({\nu }_W\right):=\frac{1}{T}\sum _{\alpha \in S}\frac{1}{N}\sum _{i\in V}{\phi }_{ik}^{\alpha +{\nu }_k}{\phi }_{i{k}^{\prime }}^{\alpha +{\nu }_{k^{\prime }}}.$$

(164)

For example, we may want to align the samples with respect to some vector ${\nu }_W^{*}$ such that ${\nu }_k^{go}\le {\nu }_k^{*}\le {\nu }_k^{mov}$ and that it maximizes the norm of the overlap matrix

$${∥\mathcal{Q}\left({\nu }_W^{*}\right)∥}_2^2=\underset{{\nu }_W}{sup}{∥\mathcal{Q}\left({\nu }_W\right)∥}_2^2,$$

(165)

in any case, notice that the minimization of such kinds of functionals could soon become impractical for large datasets. We will show an example of the rank reduction method (by renormalization) in Figure 11–13.

4.2.5. Inference Methods

Since the work of Schneidman et al. [62], the possibility of reconstructing the couplings has become a major goal in computational neuroscience, and powerful inference methods are now available [142,143,144,145]. Let us consider a relativistic theory in the mixed space $\mathcal{l}\in \mathsf{\Lambda }$ and let us consider a free field model with input kernel I. If we assume the action is that of a free (non-interacting) theory [58], then the parameters can be obtained easily from the average magnetizations by inverting the Callen equations [146,147]

$$\lambda I_{\mathcal{l}}={tanh}^{-1}\left(m_{\mathcal{l}}\right),m_{\mathcal{l}}:=\langle {\sigma }_{\mathcal{l}}\rangle ,$$

(166)

Clearly there is an error that depends on the number of samples of the empirical ensemble average. Calling n the number of experiments, or replicas, on which the ensemble average is taken (e.g., the “session trials” of [35]), the error is as follows

$$\lambda \delta I_{\mathcal{l}}=\frac{cosh\left(\lambda I_{\mathcal{l}}\right)}{\sqrt{n}}.$$

(167)

We can go further and add two body interactions, in this case the problem becomes less trivial, and we have to deal with the so-called inverse Ising problem, a classic inference problem [142,146,147,148]. Let us switch on the two body interactions $F_{\mathcal{l}{\mathcal{l}}^{\prime }}$ and introduce the “grand covariance” of the mixed space

$${\mathcal{C}}_{\mathcal{l}{\mathcal{l}}^{\prime }}:={\langle {\sigma }_{\mathcal{l}}{\sigma }_{{\mathcal{l}}^{\prime }}\rangle }_{\eta }-{\langle {\sigma }_{\mathcal{l}}\rangle }_{\eta }{\langle {\sigma }_{{\mathcal{l}}^{\prime }}\rangle }_{\eta }.$$

(168)

The values of the coupling parameters are ultimately recovered by inverting the following system of (possibly non-linear) equations [142]:

$${\mathcal{C}}_{\mathcal{l}{\mathcal{l}}^{\prime }}=\frac{\partial m_{\mathcal{l}}}{\partial I_{{\mathcal{l}}^{\prime }}}=\frac{{\partial }^2\mathsf{\Psi }}{\partial I_{\mathcal{l}}\partial I_{{\mathcal{l}}^{\prime }}},$$

(169)

that must be solved and then inverted to find the F matrix. There are various methods to do so [146,147,148], and an excellent survey is that of Nguyen et al. [142]. There are also several approximate formulas that only require us to invert the grand covariance, the simplest is

$$F_{\mathcal{l}{\mathcal{l}}^{\prime }}^{\mathrm{LR}}=-{\left({\mathcal{C}}^{-1}\right)}_{\mathcal{l}{\mathcal{l}}^{\prime }}$$

(170)

that correspond to the so-called “naive” mean-field theory [142]. More advanced formulas depending on the inverse covariance are, for example, the TAP formula [142]

$$F_{\mathcal{l}{\mathcal{l}}^{\prime }}^{\mathrm{TAP}}=\frac{2{\left({\mathcal{C}}^{-1}\right)}_{\mathcal{l}{\mathcal{l}}^{\prime }}}{1+\sqrt{1-8m_{\mathcal{l}}{m}_{{\mathcal{l}}^{\prime }}{\left({\mathcal{C}}^{-1}\right)}_{\mathcal{l}{\mathcal{l}}^{\prime }}}},$$

(171)

the “independent-pair” approximation formula [142]

$$F_{\mathcal{l}{\mathcal{l}}^{\prime }}^{\mathrm{IP}}=\frac{1}{4}ln\frac{\left[\left(1+m_{\mathcal{l}}\right)\left(1+m_{{\mathcal{l}}^{\prime }}\right)+{\mathcal{C}}_{\mathcal{l}{\mathcal{l}}^{\prime }}\right]\left[\left(1-m_{\mathcal{l}}\right)\left(1-m_{{\mathcal{l}}^{\prime }}\right)+{\mathcal{C}}_{\mathcal{l}{\mathcal{l}}^{\prime }}\right]}{\left[\left(1+m_{\mathcal{l}}\right)\left(1-m_{{\mathcal{l}}^{\prime }}\right)-{\mathcal{C}}_{\mathcal{l}{\mathcal{l}}^{\prime }}\right]\left[\left(1-m_{\mathcal{l}}\right)\left(1+m_{{\mathcal{l}}^{\prime }}\right)-{\mathcal{C}}_{\mathcal{l}{\mathcal{l}}^{\prime }}\right]},$$

(172)

and the Sessak–Monasson formula [142,149]

$$F_{\mathcal{l}{\mathcal{l}}^{\prime }}^{\mathrm{SM}}=F_{\mathcal{l}{\mathcal{l}}^{\prime }}^{\mathrm{IP}}-{\left({\mathcal{C}}^{-1}\right)}_{\mathcal{l}{\mathcal{l}}^{\prime }}-\frac{{\mathcal{C}}_{\mathcal{l}{\mathcal{l}}^{\prime }}}{\left(1-m_{\mathcal{l}}^2\right)\left(1-m_{{\mathcal{l}}^{\prime }}^2\right)-{\mathcal{C}}_{\mathcal{l}{\mathcal{l}}^{\prime }}^2},$$

(173)

especially suited in the limit of small covariances. Notice that the presence of the pair interactions also modifies the expression of the external fields. Introducing the Legendre transform of the free energy with respect to the magnetizations,

$$\mathsf{\Gamma }:=\underset{I_{\mathsf{\Lambda }}\in {\mathbb{R}}^{\mathsf{\Lambda }}}{max}\left\{I_{\mathsf{\Lambda }}·m_{\mathsf{\Lambda }}+\mathsf{\Psi }\right\},$$

(174)

the equations for both parameters are obtained from [142]

$$I_{\mathcal{l}}=\frac{\partial \mathsf{\Gamma }}{\partial m_{\mathcal{l}}},{\left({\mathcal{C}}^{-1}\right)}_{\mathcal{l}{\mathcal{l}}^{\prime }}=\frac{\partial \mathsf{\Gamma }}{\partial m_{\mathcal{l}}\partial m_{{\mathcal{l}}^{\prime }}}.$$

(175)

Using the double index, the equations for the F parameters are

$${\mathcal{C}}_{ij}^{\alpha \beta }=\frac{{\partial }^2\mathsf{\Psi }}{\partial I_i^{\alpha }\partial I_j^{\beta }}.$$

(176)

since in the non-relativistic approximation we ignored correlations where both upper and lower pairs of indices are different, these can be translated in

$${\mathcal{C}}_{ij}^{\alpha \beta }={\mathcal{C}}_{ij}^{\alpha \alpha }\mathbb{I}\left(\alpha =\beta \right)+{\mathcal{C}}_{ii}^{\alpha \beta }\mathbb{I}\left(i=j\right),$$

(177)

the relations with the elements of the covariance matrices are

$${\langle \delta c_{ij}\rangle }_{\eta }=\frac{1}{T}\sum _{\alpha \in S}{\mathcal{C}}_{ij}^{\alpha \alpha },{\langle \delta q^{\alpha \beta }\rangle }_{\eta }=\frac{1}{N}\sum _{i\in V}{\mathcal{C}}_{ii}^{\alpha \beta }.$$

(178)

This reduces from $T^2{N}^2$ to $NT(N+T)$ the number of parameters that should be actually computed to reconstruct the action, greatly enhancing the computational tractability in the non-relativistic case.

4.2.6. Renormalization

We conclude the theoretical sections with a simple renormalization [101,141] scheme, based on [56], that will be useful to link the theory with experimental observations. From this subsection we switch again to the lattice gas representation. Consider a joint kernel partition as in Section 3 of Franchini 2023 [56], with two levels (equivalent to one-step Replica Symmetry Breaking: RSB1). Let $N_1$, $N_2$, $T_1$ and $T_2$ be numbers such that $N=N_1{N}_2$ and $T=T_1{T}_2$, and let

$$\begin{array}{cc}\hfill & V_0=\left\{1\le i_1\le N_1\right\},V_{i_1}=\left\{1\le i_2\le N_2\right\},\\ \hfill & S_0=\left\{1\le {\alpha }_1\le T_1\right\},S_{{\alpha }_1}=\left\{1\le {\alpha }_2\le T_2\right\}.\end{array}$$

(179)

The kernel can be rewritten according to the new multiscale index

$$\mathsf{\Omega }=\left\{{\mathsf{\Omega }}_{i_1}^{{\alpha }_1}\in \left\{0,1\right\}:i_1\in V_0,{\alpha }_1\in S_0\right\}$$

(180)

where we introduced the sub-kernels

$${\mathsf{\Omega }}_{i_1}^{{\alpha }_1}:=\left\{{\phi }_{i_1{i}_2}^{{\alpha }_1{\alpha }_2}\in \left\{0,1\right\}:i_2\in V_{i_1},{\alpha }_2\in S_{{\alpha }_1}\right\}$$

(181)

the field is renormalized according to a map such that

$${\widehat{\phi }}_{i_1}^{{\alpha }_1}:=\mathcal{R}\left({\mathsf{\Omega }}_{i_1}^{{\alpha }_1}\right)\in \left\{0,1\right\}$$

(182)

to regain some binary variables, i.e., ${\widehat{\phi }}_{i_1}^{{\alpha }_1}$ will be one if within the cell $V_{i_1}{S}_{{\alpha }_1}$ the condition set by the renormalization map is verified, and zero otherwise. By construction, the relationship between the two variables is such that

$${\mathsf{\Omega }}_{i_1}^{{\alpha }_1}={\widehat{\phi }}_{i_1}^{{\alpha }_1}{\mathsf{\Omega }}_{i_1}^{{\alpha }_1}$$

(183)

We can define the renormalized kernel as follows:

$$\widehat{\mathsf{\Omega }}:=\left\{{\widehat{\phi }}_{i_1}^{{\alpha }_1}\in \left\{0,1\right\}:i_1\in V_0,{\alpha }_1\in S_0\right\}$$

(184)

Since the action structure is symmetrical between space and time, we can also perform the calculations on the potential term alone. We apply the multiscale index

$$\sum _{i\in V}\sum _{j\in V}{A}_{ij}\sum _{\alpha \in S}{\phi }_i^{\alpha }{\phi }_j^{\alpha }=\sum _{i_1\in V_0}\sum _{j_1\in V_0}\sum _{i_2\in V_{i_1}}\sum _{j_2\in V_{j_1}}{A}_{i_1{i}_2{j}_1{j}_2}\sum _{{\alpha }_1\in S_0}\sum _{{\alpha }_2\in S_{\alpha 1}}{\phi }_{i_1{i}_2}^{{\alpha }_1{\alpha }_2}{\phi }_{j_1{j}_2}^{{\alpha }_1{\alpha }_2}$$

(185)

and then the renormalization map

$$\sum _{i\in V}\sum _{j\in V}{A}_{ij}\sum _{\alpha \in S}{\phi }_i^{\alpha }{\phi }_j^{\alpha }=\sum _{i_1\in V_0}\sum _{j_1\in V_0}\sum _{{\alpha }_1\in S_0}{\widehat{A}}_{i_1{j}_1}^{{\alpha }_1}\left(\mathsf{\Omega }\right){\widehat{\phi }}_{i_1}^{{\alpha }_1}{\widehat{\phi }}_{j_1}^{{\alpha }_1}$$

(186)

For example, for a bin renormalization

$${\widehat{\phi }}_{i_1}^{{\alpha }_1}=\mathbb{I}({\mathsf{\Omega }}_{i_1}^{{\alpha }_1}\ne 0)$$

(187)

the effective interaction will be given by

$${\widehat{A}}_{i_1{j}_1}^{{\alpha }_1}\left(\mathsf{\Omega }\right):=\sum _{i_2\in V_{i_1}}\sum _{j_2\in V_{j_1}}{A}_{i_1{i}_2{j}_1{j}_2}\sum _{{\alpha }_2\in S_{{\alpha }_1}}{\phi }_{i_1{i}_2}^{{\alpha }_1{\alpha }_2}{\phi }_{j_1{j}_2}^{{\alpha }_1{\alpha }_2}$$

(188)

while for a renormalization by decimation (Kadanoff renormalization) [101,102,103],

$${\widehat{\phi }}_{i_1}^{{\alpha }_1}={\phi }_{i_{1}1}^{{\alpha }_{1}1}$$

(189)

we will have that

$${\widehat{A}}_{i_1{j}_1}^{{\alpha }_1}\left(\mathsf{\Omega }\right):=\sum _{i_2\in V_{i_1}\setminus \left\{1\right\}}\sum _{j_2\in V_{j_1}\setminus \left\{1\right\}}{A}_{i_1{i}_2{j}_1{j}_2}\sum _{{\alpha }_2\in S_{{\alpha }_1}\setminus \left\{1\right\}}{\phi }_{i_1{i}_2}^{{\alpha }_1{\alpha }_2}{\phi }_{j_1{j}_2}^{{\alpha }_1{\alpha }_2}$$

(190)

We separate the stationary term (if any)

$${\widehat{A}}_{i_1{j}_1}^{{\alpha }_1}\left(\mathsf{\Omega }\right):={\widehat{A}}_{i_1{j}_1}+\delta {\widehat{A}}_{i_1{j}_1}^{{\alpha }_1}\left(\mathsf{\Omega }\right)$$

(191)

The stationary term corresponds to the renormalized coupling matrix; we can thus rewrite the action potential term by separating the renormalized part from the fluctuation

$$\sum _{i\in V}\sum _{j\in V}{A}_{ij}\sum _{\alpha }{\phi }_i^{\alpha }{\phi }_j^{\alpha }=\sum _{i_1\in V_0}\sum _{j_1\in V_0}{\widehat{A}}_{i_1{j}_1}\sum _{{\alpha }_1\in S_0}{\widehat{\phi }}_{i_1}^{{\alpha }_1}{\widehat{\phi }}_{j_1}^{{\alpha }_1}+\sum _{i_1\in V_0}\sum _{j_1\in V_0}\sum _{{\alpha }_1\in S_0}\delta {\widehat{A}}_{i_1{j}_1}^{{\alpha }_1}\left(\mathsf{\Omega }\right){\widehat{\phi }}_{i_1}^{{\alpha }_1}{\widehat{\phi }}_{j_1}^{{\alpha }_1}$$

(192)

Doing the same with the kinetic term

$${\widehat{B}}_{i_1}^{{\alpha }_1{\beta }_1}\left(\mathsf{\Omega }\right):=\sum _{{\alpha }_2\in S_{{\alpha }_1}}\sum _{{\beta }_2\in S_{{\alpha }_1}}{B}^{{\alpha }_1{\alpha }_2{\beta }_1{\beta }_2}\sum _{i_2\in V_{i_1}}{\phi }_{i_1{i}_2}^{{\alpha }_1{\alpha }_2}{\phi }_{i_1{i}_2}^{{\beta }_1{\beta }_2}$$

(193)

and separating the uniform term

$${\widehat{B}}_{i_1}^{{\alpha }_1{\beta }_1}\left(\mathsf{\Omega }\right):={\widehat{B}}^{{\alpha }_1{\beta }_1}+\delta {\widehat{B}}_{i_1}^{{\alpha }_1{\beta }_1}\left(\mathsf{\Omega }\right)$$

(194)

the treatment is completely symmetrical, leading to

$$\sum _{\alpha \in S}\sum _{\beta \in S}{B}^{\alpha \beta }\sum _{i\in V}{\phi }_i^{\alpha }{\phi }_i^{\beta }=\sum _{{\alpha }_1\in S_0}\sum _{{\beta }_1\in S_0}{\widehat{B}}^{{\alpha }_1{\beta }_1}\sum _{i_1\in V_0}{\widehat{\phi }}_{i_1}^{{\alpha }_1}{\widehat{\phi }}_{i_1}^{{\beta }_1}+\sum _{{\alpha }_1\in S_0}\sum _{{\beta }_1\in S_0}\sum _{i_1\in V_0}{\widehat{B}}_{i_1}^{{\alpha }_1{\beta }_1}\left(\mathsf{\Omega }\right){\widehat{\phi }}_{i_1}^{{\alpha }_1}{\widehat{\phi }}_{i_1}^{{\beta }_1}$$

(195)

The action in the renormalized variables will therefore have a perturbation

$$\begin{array}{c}G\left(A,B\right)=\sum _{\mathsf{\Omega }\in {\left\{0,1\right\}}^{VS}}\exp \left[-\lambda \mathcal{A}\left(\mathsf{\Omega }|A,B\right)\right]=\hfill \\ =\sum _{\widehat{\mathsf{\Omega }}\in {\left\{0,1\right\}}^{V_0{S}_0}}\exp [-\lambda \mathcal{A}\left(\widehat{\mathsf{\Omega }}\right|\widehat{A},\widehat{B})-\lambda \delta \mathcal{A}\left(\widehat{\mathsf{\Omega }}\right|A,B)]=G(\widehat{A},\widehat{B}){\langle \exp [-\lambda \delta \mathcal{A}\left(\widehat{\mathsf{\Omega }}\right|A,B)]\rangle }_{\widehat{\eta }}\hfill \end{array}$$

(196)

where $\widehat{\eta }$ is the GS of the renormalized action. In general, this expression depends on the details of the couplings within the renormalized cell. The perturbation of the action is formally defined as

$$\delta \mathcal{A}\left(\widehat{\mathsf{\Omega }}\right|A,B):=-\frac{1}{\lambda }log\sum _{\mathsf{\Omega }\in \mathcal{K}\left(\widehat{\mathsf{\Omega }}\right)}\exp [-\lambda \mathsf{\Gamma }(\mathsf{\Omega },\widehat{\mathsf{\Omega }}|A,B)]$$

(197)

where the sum is on those $\mathsf{\Omega }$ that if renormalized are equal to $\widehat{\mathsf{\Omega }}$, i.e.,

$$\mathcal{K}\left(\widehat{\mathsf{\Omega }}\right):=\{\mathsf{\Omega }\in {\left\{0,1\right\}}^{VS}:\mathcal{R}\left(\mathsf{\Omega }\right)=\widehat{\mathsf{\Omega }}\}$$

(198)

and the function $\mathsf{\Gamma }$ is defined as follows:

$$\mathsf{\Gamma }(\mathsf{\Omega },\widehat{\mathsf{\Omega }}|A,B):=\sum _{i_1\in V_0}\sum _{j_1\in V_0}\sum _{{\alpha }_1\in S_0}\delta {\widehat{A}}_{i_1{j}_1}^{{\alpha }_1}\left(\mathsf{\Omega }\right){\widehat{\phi }}_{i_1}^{{\alpha }_1}{\widehat{\phi }}_{j_1}^{{\alpha }_1}+\sum _{{\alpha }_1\in S_0}\sum _{{\beta }_1\in S_0}\sum _{i_1\in V_0}\delta {\widehat{B}}_{i_1}^{{\alpha }_1{\beta }_1}\left(\mathsf{\Omega }\right){\widehat{\phi }}_{i_1}^{{\alpha }_1}{\widehat{\phi }}_{i_1}^{{\beta }_1}$$

(199)

Thus, renormalization operations can also change the structure of the action. For example, consider the potential part: we can approximate the renormalized coupling fluctuations with a stationary Random Energy Model (REM universality, see Arous and Kuptsov [150] or Section 6 of Franchini 2023 [56,58] for a practical example in kernel language).

$$\delta {\widehat{A}}_{i_1{j}_1}^{{\alpha }_1}\left(\mathsf{\Omega }\right)\approx J_{i_1{j}_1}\left(\mathsf{\Omega }\right)\sqrt{{\mathsf{\Delta }}_{i_1{j}_1}}.$$

(200)

The partition function can be approximated as follows:

$$\begin{array}{c}\sum _{\mathsf{\Omega }\in \mathcal{K}\left(\widehat{\mathsf{\Omega }}\right)}\exp \left[-\lambda \sum _{i_1\in V_0}\sum _{j_1\in V_0}\sum _{{\alpha }_1\in S_0}\delta {\widehat{A}}_{i_1{j}_1}^{{\alpha }_1}\left(\mathsf{\Omega }\right){\widehat{\phi }}_{i_1}^{{\alpha }_1}{\widehat{\phi }}_{j_1}^{{\alpha }_1}\right]\approx \hfill \\ \approx \sum _{\mathsf{\Omega }\in \mathcal{K}\left(\widehat{\mathsf{\Omega }}\right)}\exp \left[-\lambda \sum _{i_1\in V_0}\sum _{j_1\in V_0}{J}_{i_1{j}_1}\left(\mathsf{\Omega }\right)\sqrt{{\mathsf{\Delta }}_{i_1{j}_1}}\sum _{{\alpha }_1\in S_0}{\widehat{\phi }}_{i_1}^{{\alpha }_1}{\widehat{\phi }}_{j_1}^{{\alpha }_1}\right]=\hfill \\ =\exp \left({\widehat{\lambda }}^2{T}_1^2\sum _{i_1\in V_0}\sum _{j_1\in V_0}{\mathsf{\Delta }}_{i_1{j}_1}{\widehat{\varphi }}_{i_1{j}_1}^2\right)\hfill \end{array}$$

(201)

where in the second row we applied the PPP-REM [56] average and $\widehat{\lambda }$ is the renormalized temperature. In essence, this type of mean field approximation introduces a linear term in the renormalization map

$$A_{ij}\to {\widehat{A}}_{i_1{j}_1}-\widehat{\lambda }{T}_1{\mathsf{\Delta }}_{i_1{j}_1}{\widehat{\varphi }}_{i_1{j}_1}+\cdots$$

(202)

which results in a quadratic term added to the action

$$\sum _{i\in V}\sum _{j\in V}{A}_{ij}{\varphi }_{ij}\to \sum _{i_1\in V_0}\sum _{j_1\in V_0}{\widehat{A}}_{i_1{j}_1}{\widehat{\varphi }}_{i_1{j}_1}-\widehat{\lambda }{T}_1\sum _{i_1\in V_0}\sum _{j_1\in V_0}{\mathsf{\Delta }}_{i_1{j}_1}{\widehat{\varphi }}_{i_1{j}_1}^2+\cdots$$

(203)

Then, in the first approximation we could ignore the corrections terms in the PMd experiments with Utah96, due to the small magnitude of the correlations. Notice that this could also explain other deviations from the max entropy principle like those shown in Figure 2 of Meshulam et al. [65]. More accurate renormalization schemes based on multi-scale analysis can be computed following the methods of Franchini 2023 [56,57,58] and many other methods as well [52,101,102,103,107], although in general the exact shape of the perturbations depend on the details of the system and on the instrumental limits and systematics, and to push further it is therefore necessary to introduce more specified information about the couplings and the kinetic properties of the system, both of the neocortex and the sensor.

5. Experimental Methods

5.1. Cortical Minitubes

So far, the most accepted theory for the anatomical and functional organization of the retina is the columnar model (see Figure 3) [88,91,92], and similar assemblies of neurons are observable trough the whole neocortex, at least at the anatomical level. Anyway, since in the retina there is also a well-established corresponding functional organization that has still not been shown for the whole neocortex, in the following we will use the name “minitubes” to indicate only the anatomical structures that are seen from a histological inspection (Figure 5). Then, let ${\mathbb{L}}_3$ be a cubic lattice and let $xyz\in {\mathbb{L}}_3$ such that z represents, for example, the average height from the surface of the cortex at which a given cortical layer is located.

Decimated Kernel

Let $xy$ be the position of the center of gravity of the cortical minitube section in the horizontal plane. To model the minitube layers we will define a partition of the space ${\mathbb{R}}^3$ into volumes of equal size according to the lattice cells; for simplicity, we will approximate the cortical minitubes with square-based minitubes. Notice that the present charting of the neocortex is not accounting for the neural connections that may have any topology and are encoded in the interaction matrix A. The reason for using a euclidean reference frame is to allow for comparisons with existing histologies and fMRIs, as well as other data [151]. Also, it may highlight effects due to possible extracellular fields and currents [152], whose correlations may follow a euclidean topology. The layers of the minitubes are thus represented by the lattice cells

$$U_{xyz}:=U_x{U}_y{U}_z\subset {\mathbb{R}}^3$$

(204)

Now, calling $v\left(i\right)\in {\mathbb{R}}^3$ the position of the nucleus of the i-th neuron, we can group by the volume in which they are located

$$V_{xyz}:=\left\{i\in V:v\left(i\right)\in U_{xyz}\right\}$$

(205)

each of these groups of neurons will have its own associated kernel

$${\mathsf{\Omega }}_{xyz}:=\left\{{\phi }_i^{\alpha }\in \left\{0,1\right\}:i\in V_{xyz},\alpha \in S\right\}.$$

(206)

At this point, one could further group the neurons, first by index z, so as to form the cortical minitubes. The vertices belonging to the minitube are

$$V_{xy}:=\bigcup _{z\in \mathbb{L}}{V}_{xyz}$$

(207)

that is the set of neurons that constitutes the minitube at position $xy$. The kernel is

$${\mathsf{\Omega }}_{xy}:=\{{\mathsf{\Omega }}_{xyz}\in {\left\{0,1\right\}}^{V_{xyz}}:z\in \mathbb{L},\alpha \in S\}$$

(208)

and describes the activity of the single cortical minitube in $xy$. Some interfaces, such as Neuropixel or deep multielectrode shanks, allow direct observations of this activity. The minitubes are in the end grouped again to form the cortex structures and areas,

$$V:=\bigcup _{xy\in {\mathbb{L}}_2}{V}_{xy}$$

(209)

and the original kernel can thus be expressed in terms of the minitubes:

$$\mathsf{\Omega }=\{{\mathsf{\Omega }}_{xy}^{\alpha }\in {\left\{0,1\right\}}^{V_{xy}}:xy\in {\mathbb{L}}_2,\alpha \in S\},$$

(210)

so that it represents a two-dimensional lattice of cortical minitubes [88,91,92,153,154,155], a system in 2 + 1 + 1 dimensions. For the above, we can consider the experimental kernel for a specific tubular layer

$${\mathsf{\Omega }}_z:=\{{\mathsf{\Omega }}_{xyz}^{\alpha }\in {\left\{0,1\right\}}^{V_{xyz}}:xy\in {\mathbb{L}}_2,\alpha \in S\},$$

(211)

where, again, $xyz\in {\mathbb{L}}_3$ are the spatial coordinates in a cubic lattice such that z represents the average height from the surface of the cortex at which a given layer is located, and $xy$ is the position of the minitube section in the horizontal plane. The points are organized in a planar sub-lattice $x^{\prime }{y}^{\prime }\in {\mathbb{L}}_2^{\prime }$ (of the observed cortical layer z) whose step is much greater than the diameter of the individual minitube, so that the activities recorded at the various points belong with high probability to different and well-spaced minitubes. At this point, to model the spacing between the probing points, we apply a renormalization by decimation on $\mathsf{\Omega }$ and obtain the decimated activity kernel

$$\widehat{\mathsf{\Omega }}:=\{{\widehat{\phi }}_{x^{\prime }{y}^{\prime }}^{\alpha }\in \left\{0,1\right\}:x^{\prime }{y}^{\prime }\in {\mathbb{L}}_2^{\prime },\alpha \in S\},{\widehat{\phi }}_{x^{\prime }{y}^{\prime }}^{\alpha }:=\mathbb{I}({\mathsf{\Omega }}_{x^{\prime }{y}^{\prime }z}^{\alpha }\ne 0).$$

(212)

this is the electrode kernel of Equation (25) shown in the main text. This kernel is intended to model approximately the sensor recording, net of systematic errors and approximations. According to our arguments it should be comparable with a renormalized theory. Notice that this renormalization happens only in space and hence the information coming from the digitalization of neuronal signals is largely preserved (as far as the signals inside a channel or multi-units do not overlap too much in time). $\widehat{\mathsf{\Omega }}$ leads to the experimental hypermatrix of Figure 1. Finally, notice that the present charting of the neocortex is not accounting for the anatomical neural connections that may have any topology and are encoded in the interaction matrix A. Clearly, determining the exact effective theory that can describe the dynamics of the columns and their excitations will require careful analysis of the body of knowledge about the structure of the neocortex and the interface itself, but these manipulations demonstrate that a treatment in terms of field theory is possible, at least in this formalism. Moreover, given the particular architecture of the cortex, it is possible that the topology of such a theory is essentially either mean-field or two-dimensional, and with layers of cortices behaving as interacting fields, just as in elementary particle theory. This could greatly facilitate the analytical construction of effective theories.

5.2. Neural Recordings with Utah 96

5.2.1. Subjects

Two male rhesus macaque monkeys (Macaca mulatta, Monkeys P and C), weighing 9 and 9.5 kg, respectively, were employed for the task shown as the case study. Animal care, housing, surgical procedures and experiments conformed to European (Directive 86/609/ECC and 2010/63/UE) and Italian (D.L. 116/92 and D.L. 26/2014) laws and were approved by the Italian Ministry of Health. Monkeys were pair-housed with cage enrichment. They were fed daily with standard primate chow that was supplemented with nuts and fresh fruits if necessary. During recording days, the monkeys received their daily water supply during the experiments.

5.2.2. Apparatus and Task

The monkeys were seated in front of a black isoluminant background (<0.1 cd/m²) of a 17-inch touchscreen monitor (LCD, 800 × 600 resolution), inside a darkened, acoustic-insulated room. A non-commercial software package, CORTEX (http://www.nimh.gov.it, accessed on 1 January 2010), was used to control the presentation of the stimuli and the behavioural responses. Figure 1 and Figure 6 panel C show the scheme of the task: a Go-signal reaching task. Each trial started with the appearance of a central target (CT) (red circle, diameter 1.9 cm). The monkeys had to reach and hold the CT. After a variable holding time (400–900 ms, 100 ms increments) a peripheral target (PT) (red circle, diameter 1.9 cm) appeared randomly in one of two possible locations (right/left, D1/D2) and the CT disappeared (Go signal). After the Go signal the subjects had to reach and hold the PT for a variable time (400–800 ms, 100 ms increments) to receive juice. The time between the presentation of the Go signal and the onset of the hand movement (M_on) is the reaction time (RT). White circles around the central target were used as feedback for the animals to indicate the touch.

5.2.3. Extraction and Processing of Neuronal Data

A multielectrode array (Blackrock Microsystems, Salt Lake City) with 96 electrodes (Utah 96, spacing 0.4 mm) was surgically implanted in the left dorsal premotor cortex (PMd; the references used after opening the dura were the arcuate sulcus and pre-central dimple) to acquire unfiltered electric field potentials (UFP; i.e., the raw signal) sampled at 24.4 kHz (Tucker Davis Technologies, Alachua, FL, USA). As described in previous work from our group [35], we extracted single neurons activities from the raw signal by employing the spike sorting toolbox KiloSort3 [156] with the following parameters. Thresholds: [9 9] (thresholds for template-matching on spike detection); Lambda: 10 (bias factor of the individual spike amplitude towards the cluster mean); Area Under the Curve split: 0.9 (threshold for cluster splitting); and number of blocks: 5 (amount of blocks channels are divided into for estimating probe drift). The output was manually curated in Phy (v2.0; 17) to merge clusters that were mistakenly separated by the automated sorter. From this procedure we obtained a binary spike raster with a time resolution of 1ms (1 for a spike, 0 for no spikes) for each single trial of the experiment. Each single-trial raster was then put into the form of the kernel $\widehat{\mathsf{\Omega }}$ of Equation (212).

5.2.4. Neural Dynamics Underlying Movement Generation in PMd

We chose this task as a use-case for its simplicity as it involves only two experimental conditions. In this way, the results obtained in our LFT context are directly comparable with those obtained previously using common approaches that rely on covariance analysis [35,99,157,158,159,160]. We extracted the kernel $\langle \widehat{\mathsf{\Omega }}\rangle$ in relation to the movement onset (M_on), considering an epoch of 1s before and after the event. By doing so, the distributions of the behavioral events of the task (the Go signal and M_on) are included (see Figure 1). It has been demonstrated that, during the time preceding the movement, PMd neurons express strong modulations associated with movement control [35,86,108,158,161,162]. The hypermatrices computed for the two experimental conditions are shown in Figure 7, Figure 8, Figure 9 and Figure 10. The JS matrices exhibit striking features, and by comparing them across movement directions, one can retrieve most of the hallmarks of PMd neural dynamics. The first is the strong increase in synchronous activity peaking within the 200 ms interval preceding the M_on (black markers in Figure 1) that correspond to the functional state of the system linked to the incoming movement generation. Indeed, the motor planning of actions in PMd is recognized to be encoded at the population level in the form of synchronization patterns that exhibit a strong modulation around 200 ms before the onset of movement [35,77,78,82,86,108,109,110,157,158,159,161,162,163,164,165,166,167,168,169,170,171,172,173,173]. The second is the specificity of PMd neurons for the direction of movement, which in the reported task could happen towards the left or right (D1/D2). In Figure 1 and Figure 8 (ED), this is evidenced by the more intense motifs of synchrony for one direction (D2) with respect to the other (D1). They emerge at the end of the motor plan maturation (∼ within 200 ms before M_on), continuing for at least 200 ms afterwards. In the ED section we report examples from a second subject and, separately, the components of the hypermatrix with additional details (e.g., the difference |D1-D2| for both $\langle \widehat{\mathsf{\Pi }}\rangle$ and $\langle \widehat{\mathsf{\Phi }}\rangle$.) Significantly, the dynamic contributions detectable from the JS matrix can be easily mapped in the spatial domain thanks to the hypermatrix arrangement, which emphasize the correspondences between the JS matrix, the kernels $\langle \widehat{\mathsf{\Omega }}\rangle$ and the spatial and temporal averages. For example, from the kernels in Figure 1 and the zoom of Figure 8, the firing patterns that elicit a specific configuration of dynamical synchrony can be identified. This reveals that the temporal correlations during the motor plan maturation are caused by a specific firing sequence in the kernel $\langle \widehat{\mathsf{\Omega }}\rangle$ (for both D1 and D2). Hence, we can infer that the maturation of the motor plan corresponds to different populations of neurons discharging with variable timings and intensities for D1 compared to D2. The JS matrix also demonstrates that the direction-specific correlations coincide with more intense firing for D2 compared to D1. In addition to direction-specific differences, relevant similarities are also appreciable. The cross-emerging at the center of the JS matrix represents synchronization among neural ensembles that extends throughout the duration of the trial for both D1 and D2. Again, the neural assemblies responsible can be easily identified from the kernels $\langle \widehat{\mathsf{\Omega }}\rangle$. Future work will be needed to clarify more details. The spatial correlations are instead recoverable from the matrix $\langle \widehat{\mathsf{\Phi }}\rangle$. In our example, it can be noted how the combinations underlying the motor plan are preserved for both directions (same correlation values in $\langle \widehat{\mathsf{\Phi }}\rangle$ for both directions), while the direction-specific ones change.

5.2.5. Comparison with Other Methods

Thus, with the hypermatrix representation, neural dynamics can be efficiently decomposed into their spatial and temporal contributions, and their roles in the studied task are easily mapped. From these remarks, we understand the striking traits of the hypermatrix: its completeness despite its simplicity. It conveys fundamental information about the system in a compact representation without the need for complex numerical artifice. This is a substantial difference with other approaches frequently used to analyze neural activity (e.g., PCA or machine learning methods among the most popular [24,35,77,78,79,80,81,82]). Although these methods have provided valuable insights, none of them offer a picture encompassing the temporal and the spatial attributes of the system at the same time. Moreover, the connection these methods make between recorded activity and circuit mechanisms is elusive and hardly generalizable. In the case of PCA, for example, the temporal and spatial properties can be linked together only after a non-trivial, and most of the time arbitrary, sequence of numerical steps. Among others, these include a dimensionality reduction, i.e., choosing a number of PCs and the subsequent projections onto the reduced space; this requires computing the eigenvectors of the covariance matrix. In contrast, our theory only requires simple scalar products of the experimental rasters, eliminating the need for dimensionality reduction. In addition, the interpretations that conventional methods offer about the intrinsic nature of neural processes are strongly dependent on the chosen analysis pipeline and are far from being derived from the universal principles of a physical theory. This significantly impacts, for example, the definition that these methods can provide for the energy of the system, which remains vague and unformalized (such as in the case of the manifold hypothesis [174] and the widespread PCA-based energy landscapes [79,174,175,176]. We have instead shown that the kernel, its transpose and the corresponding scalar products give an accurate and physical-based description of the energy functional of the system. Most importantly, our approach entails a formal communication between physics and neuroscience using as a language the governing equations of elementary particles. This allows for the measurement of neural interactions through physically grounded observables and their interpretation in terms of well-known laws. In our LFT framework, temporal and spatial correlations have a precise meaning, representing, respectively, the kinetic and potential energy terms of the recorded neurons. As detailed in Section 4, our energy functional is obtained through the parameters of the theories A, B and I. See Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17.

5.2.6. Test of the (Renormalized) Neural LFT

Generally speaking, the first requirement of a theory is that it should be possible to estimate the variables that describe it from experimental data (more formally, inverting the model). In our case, the set A, B and I may be recovered by inverting the hypermatrix. To do so, it is necessary to resort to a class of well-defined methods that go by the name of inverse Ising problems [142,177]. The same class of methods have been used by Tkacik et al. [63] to estimate the couplings of the Ising Hamiltonian with which they modeled the salamander retina recordings. This could also apply to the use-case here discussed, but at the price of a remarkable computational burden, mostly due to the very high rank of the JS matrix. To this respect, a viable way to lighten it could be to properly bin (renormalize) the process according to a larger clock time $\tau$. This would yield a JS matrix of a smaller rank without losing too much information. Following these considerations, we applied a bin renormalization to the kernel on a time step of 10 ms (at the level of individual trial), reducing by a factor of 100 the number of kernel cells to deal with. From Figure 18, Figure 19, Figure 20 and Figure 21 it is evident that the kernel and the patterns in the covariance matrices are almost unaffected by the chosen renormalization, at least for this type of behavioral task. We were able to compute the grand covariance of the renormalized kernel: the distributions of the matrix entries are shown in Figure 21. We see that the red distribution follows the expected normal product peak centered on zero due to the product of independent Gaussian fluctuations, which is also in the blue and green distributions. But, notice that the most correlated pairs deviating from the normal product distribution are only in blue and green. This shows that if we ignore correlation below a certain threshold (which in this case is around 5%) then we can approximate the activity with the simplified “non-relativistic” action proposed in this paper. Notice that the deviations contributing to the overlap matrix are still much smaller than those contributing to the correlation matrix and should produce only small deviations from the max entropy model of Schneidman et al. [62].

5.3. Perspectives

5.3.1. Microscopic Models

It has been proposed that a movement may be carried out by the suppression of some steady signal that ends the holding or “non-movement state” and triggers the movement [35]. This idea is in line with the shared view whereby a command initiated in other regions is executed locally in the PMd, which is part of a larger network subserving motor control based on frontal, parietal, subcortical, cerebellar and spinal structures. According to our formalism, we can state that the part of the brain deciding the movement sends the command to the PMd in the form of a spatially structured external field that is stationary throughout the execution of the computation. In an analogy with magnetic systems, such an external field configures the phase toward which the population of neurons will try to balance. It can be hypothesized that the neural computation underlying the so-called motor plan is performed in convergence to the system’s equilibrium: at the time $\alpha$ in which the external input changes, the system converges to the phase (valley) selected by the new input. This can be modeled with the magnetization profile of a one-dimensional Ising chain subject to some external field. If the field is suddenly switched on at time ${\alpha }_0$, the Lagrangian contains a one-dimensional Ising kinetic term in ${\alpha }_0$: this is to force the stationary dynamic with an average interspike period $\tau$ that is deduced from a time covariance matrix ${\langle \delta Q\rangle }_{\eta }$ (see Figures in Section 5). This simple interface model in one dimension was introduced and solved by Robert and Widom in [178] adapting methods from Percus, Tejero and others [179,180,181]. One can confront the shape of the transient field with that predicted by [178]. This mechanism also sets the typical relaxation timescale of the process. In this scenario it would be possible to construct analytically solvable models with a locally stationary external input, like the aforementioned model, which could faithfully represent local circuitry. For example, one could formally model the circuit sketched in Pani et al. [35] and check it against experimental data. It would also be possible to directly apply the “layer representation” (a repeated application of the Bayes rule) introduced in Franchini 2021 [57,58] and Franchini 2023 [56] to compute the partition function associated to the action of various deep (layered/hierarchical) models, like the synfire of Moshe Abels et al.

5.3.2. Movement and the Glassy Phase

Like the salamander retina, the PMd (ora other cortices) might also be structurally capable of exhibiting glassy phases, however, it is not necessarily the case that these are physiologically within the “computation” of movement, nor that they play a central role in sending the system off balance (at least until consciousness is in play). For example, unlike the retina, which is a structure strictly devoted to “inputs” to be passed to the central nervous system (which in the case of [63] is also detached from it), we recorded from a system that should mainly process and produce an “output” to the muscles or other areas. If the neural system responsible for movement is in a glassy phase (not going to equilibrium quickly), it might be unable to consistently convey motor commands. As a result, the executed movements may deviate from the intended actions of the animal, leading to inaccuracies such as missing the targets or unintentional actions. Moreover, the time covariance matrix (see Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21) supports the idea that the movement is not glassy: the overlap covariance matrix does not contain the movement (except in the refractory profile) and the firing patterns of the neuron are consistent with a noise model of the kind considered in [58]. Notice that the refractory period induces a structure in the overlap covariance matrix that is approximately stationary, and that the approximate symmetry of the overlap between trials shows that replica symmetry is only slightly broken. Following the ideas of [18] that see the learning as a selection of possible states of the system, we would expect more “glassy” behavior during the initial stages of training, when the monkey still has not entirely learned the task requirements. This could be studied by calculating, for example, the overlap between kernels of sessions separated by large time intervals, but the known degradation problems of Utah interfaces could mask fine-grained differences. Also, it could be possible that glassy activity may insurge in similar conditions as those considered for ex vivo salamander retina. For example, it would be of extreme interest to study the exceptionally rare recording of a dying brain published in [182], which is from the same monkey studied in [35].

5.3.3. Computing Physical LFTs with Brain Organoids

In addition to the orthodox purpose of reading and interpreting activity of natural neural networks in vivo, even more interesting applications have been made possible from recent advances in growing, shaping and interfacing biological neural tissue. The most striking example is perhaps the digital interfacing of brain organoids [183,184], a method that has already reached a fairly good technical level as demonstrated in T. Sharf et al., 2022 [184]. In short, brain organoid modeling is an advanced technique for studying brain development, physiology, function and disease occurrence (see Zheng et al., 2022 [183] review for an interesting overview). The experimental possibilities in this regard would certainly be of far reach, less expensive on both ethical and material sides and would also provide a safer guide for studying animal and human brains in vivo. There are now concrete possibilities of building hybrid circuits by connecting artificial neural networks and brain organoids [183] through currently available interfaces that could then be trained in the binary LFT language. Also, natural neural networks have been shown to work on a more efficient energetic basis and to learn from fewer examples. For example, shaping natural neural networks into useful neural circuitry [183] may allow us to realize in practice the ideas described in [51] and use natural neurons to run physical LFT simulations.