From Boltzmann to Zipf through Shannon and Jaynes

Abstract

The word-frequency distribution provides the fundamental building blocks that generate discourse in natural language. It is well known, from empirical evidence, that the word-frequency distribution of almost any text is described by Zipf’s law, at least approximately. Following Stephens and Bialek (2010), we interpret the frequency of any word as arising from the interaction potentials between its constituent letters. Indeed, Jaynes’ maximum-entropy principle, with the constrains given by every empirical two-letter marginal distribution, leads to a Boltzmann distribution for word probabilities, with an energy-like function given by the sum of the all-to-all pairwise (two-letter) potentials. The so-called improved iterative-scaling algorithm allows us finding the potentials from the empirical two-letter marginals. We considerably extend Stephens and Bialek’s results, applying this formalism to words with length of up to six letters from the English subset of the recently created Standardized Project Gutenberg Corpus. We find that the model is able to reproduce Zipf’s law, but with some limitations: the general Zipf’s power-law regime is obtained, but the probability of individual words shows considerable scattering. In this way, a pure statistical-physics framework is used to describe the probabilities of words. As a by-product, we find that both the empirical two-letter marginal distributions and the interaction-potential distributions follow well-defined statistical laws.

1. Introduction

Zipf’s law is a pattern that emerges in many complex systems composed by individual elements that can be grouped into different classes or types [1]. It has been reported in demography, with citizens linked to the city or village where they live [2]; in sociology, with believers gathering into religions [3]; in economy, with employees hired by companies [4]; and also in ecology [5,6], communications [3,7], cell biology [8], and even music [9,10,11]. In all these cases, the size of the groups in terms of the number of its constituent elements shows extremely large variability, more or less well described in some range of sizes by a power-law distribution with an exponent close to two (for the probability mass function; this turns out to be an exponent close to one for the complementary cumulative distribution).

Of particular interest is Zipf’s law in linguistics [12,13,14,15,16,17], for which individual elements are word tokens (i.e., word occurrences in a text), and classes or groups are the words themselves (word types). In this way, the “size” of a word type is given by the number of tokens of it that appear in the text under study (in other words, the absolute frequency of the word), and thus, the linguistic version of the law states that the frequency of word types can be described by a power-law probability mass function, with an exponent around two. Some variability has been found in the value of the exponent regarding the language [18] or age of the speakers [19], but not the length of the text [20,21] or the precise word definition (i.e., word forms versus lemmas [20]). Let us clarify that (what we call today) Zipf’s law was discovered more than 100 years ago by Estoup [22] and formalized mathematically by the recognized physicist E. Condon in 1928 [23]; the posterior rediscovery and intensive work by Zipf [24] is what made the law so well-known.

There have been many attempts to provide a mechanism for this curious law [25,26,27]. With text generation in mind, we can mention monkey typing, also called intermittent silence [28] (criticized in [29]), the least effort principle [30,31,32], sample-space reduction [33,34], and codification optimization [35]. More general mechanistic models for Zipf’s law are preferential attachment [18,36,37,38], birth-and-death processes [39], variations of Polya urns [40] and random walks on networks [41]. The existence of so-many models and explanations is a clear indication of the controversial origin of the law. Furthermore, there have been also important attempts to explain not only Zipf’s law but any sort of power-law distributions in nature [42,43,44,45].

A different approach is provided by the maximum-entropy principle. In statistical physics it is well known that a closed system in equilibrium with a thermal bath displays fluctuations in its energy but keeping a constant mean energy. As Jaynes showed [46], the maximization of the Shannon entropy [47] with the constrain that the mean energy is fixed yields the Boltzmann factor, which states that the probability of any microstate has to be an exponential function of its energy (note that this does not mean that the distribution of energy is exponential, as the number of microstates as a function of the energy is not necessarily constant).

Therefore, some authors have looked for an analogous of the Boltzmann factor for power laws. For example, one can easily obtain a power law not imposing a constant (arithmetic) mean but a constant geometric mean [48] (assuming also a degeneracy that is constant with respect the energy). Also, fixing both the arithmetic and the geometric mean leads to a power law with an exponential tail [49]. Nevertheless, the physical meaning of these constraints is difficult to justify.

More recently, Peterson et al. [50] have proposed a concrete non-extensive energy function that leads to power-law tails of sizes when maximizing the Shannon entropy. The main idea is that the probability is exponential with the energy, but the energy is logarithmic with size, resulting in an overall power law for sizes [50]. Other authors have found the use of the Shannon entropy inadequate, due to its close connection with exponential distributions, and have generalized the very entropy concept, yielding non-extensive entropies such as the Havrda-Charvát entropies [51], also called Tsallis entropies [52], and the Hanel-Thurner entropies [53,54].

Here we will follow the distinct approach of Stephens and Bialek [55], extending their results. Like Peterson et al. [50], these authors [55] consider the well-known Jaynes’ maximization of the plain Shannon entropy, but in contrast to them [50], no functional form is proposed a priori for the energy. Instead, the constrains are provided by the empirical two-body marginal distributions. The framework is that of word occurrence in texts, and words are considered to be composed by letters that interact all to all, in pairs. In a physical analogy, one could think of a word as a (one-dimensional) molecule, and the constituent letters would be the corresponding atoms. The interaction between atoms (letters) does not only depend on the distance but also on the position (i.e., the interaction between a and b is not the same between positions 1 and 3 than between 2 and 4, and so on; moreover, symmetry is not preserved). Let us remark that this is different from a Markov model [47]; all-to-all interaction is an important distinction. The resulting Boltzmann-like factor allows one to identify, in a natural way, the Lagrange multipliers (obtained in the maximization of entropy under the empirical values of the constrains) with the interaction potentials (with a negative sign).

Stephens and Bialek [55] only considered four-letter English words and performed a visual comparison with the empirical frequencies of words. We will considerably extend their results by analyzing words of any length from 1 to 6 letters in a much larger English corpus, and will undertake a quantitative statistical analysis of the fulfillment of Zipf’s law. In this way, using Shannon and Jaynes’ framework we will obtain a Boltzmann-like factor for the word probabilities that will allow a direct comparison with Zipf’s law. We will pay special attention to the values of the interaction potentials. The main conclusion is that two-body (two-letter) pairwise interactions are able to reproduce a power-law regime for the probabilities of words (which is the hallmark of Zipf’s law), but with considerable scatter of the concrete values of the probabilities.

In the next section, we review the maximum-entropy formalism and its application to pairwise interaction of letters in words, using the useful concept of feature functions. Next, we describe the empirical data we use and the results, including the empirical pairwise marginals (which are the input of the procedure) and the resulting pairwise potentials (which are the output from which the theoretical word distribution is built). The Zipfian character of the theoretical word distribution as well as its correspondence with the empirical distribution is evaluated. In the final section we discuss limitations and extensions of this work.

2. Maximum Entropy and Pairwise Interactions

“Information theory provides a constructive criterion for setting up probability distributions on the basis of partial knowledge,” which leads to a special type of statistical inference. This is the key idea of Jaynes’ maximum-entropy principle [46]. The recipe can be summarized as: use that probability distribution which has maximum Shannon entropy subject to whatever is known. In other words, everything should be made as random as possible, but no more [56] (E. G. Altmann has made us notice that Jaynes, being close to be a Bayesian, would not have totally agreed with the identification of entropy with randomness, and would have prefer the use of “ignorance”. Therefore, we could write instead: we should be as ignorant as possible, but no more.).

Let us consider words in texts. Labelling each word type by j, with $j=1,2,\dots V$, and V the size of the vocabulary (the total number of word types), the Shannon entropy is

$$S=-\sum _{j=1}^V{P}_{j}ln{P}_j,$$

where $P_j$ is the probability of occurrence of word type j. Please note that as we use natural logarithms, the entropy is not measured in bits but in nats, in principle. To maximize the entropy under a series of constrains one uses the method of Lagrange multipliers, where one finds the solution of

$$\frac{\partial \mathcal{L}}{\partial P_j}=-ln{P}_j-1-\alpha \frac{\partial }{\partial P_j}\left(\mathrm{constrain}1\right)-\beta \frac{\partial }{\partial P_j}\left(\mathrm{constrain}2\right)-\cdots =0,$$

(1)

for all j, with $\alpha$, $\beta$, etc., the Lagrange multipliers associated with constrain 1, constrain 2, etc., and $\mathcal{L}=S-\alpha \times \left(\mathrm{constrain}1\right)-\beta \times \left(\mathrm{constrain}2\right)-\dots$ the Lagrangian function.

One can see that the maximum-entropy method yields intuitive solutions in very simple cases. For example, if no constrains are provided one obtains the equiprobability case, $P_j^{\mu c}=1/V$ (as there is in fact one implicit constrain: normalization; $\mu c$ stands from microcanonical, in analogy with statistical physics). If there are no other constrains it is clear one cannot escape this “rudimentary” solution. If, instead, one uses all empirical values as constrains, one gets the same one puts, with a solution $P_j^{full}=\rho \left(j\right)$, with $\rho \left(j\right)$ the empirical probability of occurrence of word j (i.e., the relative frequency of j). Therefore, the full data is the solution, which is of little practical interest, as this model lacks generalization and does not bring any understanding.

More interestingly, when the mean value of the energy is used as a constrain (as it happens in thermodynamics for closed systems in thermal equilibrium with a bath), the solution is given by the Boltzmann distribution [57],

$$P_j^{can}=\frac{e^{-\beta E_j}}{Z},$$

(2)

with the notation $can$ coming from the analogy with the canonical ensemble, $E_j$ referring to the energy of type (or state) j, and with $Z={\sum }_j{e}^{-\beta E_j}$. If one could propose, a priori, an expression for the energy $E_j$ of a word type, the word probability would follow immediately; however, that energy would lack interpretation. Let us stress that we are not referring to the physical (acoustic) energy [58,59,60], as arising in speech; in fact, our approach (which is that of Stephens and Bialek) would lead to something analogous to the energy of a word. However, the analogy one finds in this way (through the Boltzmann factor) is so neat that it is not possible to escape identifying that with a sort of “energy”.

2.1. Feature Functions and Marginal Probabilities

At this point it becomes useful to introduce the feature functions [61]. Given a feature i, the feature function $f_i\left(j\right)$ is a function that for each word j takes the values

$$f_i\left(j\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if} \mathrm{the} \mathrm{word} j \mathrm{contains} \mathrm{feature} i\hfill \\ 0\hfill & \mathrm{if} \mathrm{not}\hfill \end{array}\right.$$

For example, let us consider the feature $i=\{$ letter c is in position $1\}$, summarized as $i=1c$ (in this case, this could be called letter function); then $f_{1\mathtt{c}}\left(\mathtt{cat}\right)=1$ and $f_{1\mathtt{c}}\left(\mathtt{mice}\right)=0$, as c is the first letter in cat but not in mice (let us mention that, for us, capital and lower-case letters are considered the same letter).

Considering m features, each one yielding a constrain for its expected value, we have

$$\langle f_i\rangle =\sum _{j=1}^V{P}_j{f}_i\left(j\right)=\sum _{js.t.iinj}{P}_j=F_i$$

(3)

for $i=1,2,\dots m$, with $F_i$ the empirical mean value of feature i (fraction of word tokens with feature i). Please note that $P_j$ and $\langle f_i\rangle$ are unknown, whereas $F_i$ should not. With these m constrains, the method of Lagrange multipliers [Equation (1)] leads to

$$\frac{\partial \mathcal{L}}{\partial P_j}=-ln{P}_j-1+\sum _{i=1}^m{\lambda }_i{f}_i\left(j\right)=0,$$

where ${\lambda }_i$ are now the Lagrange multipliers (we have in fact inverted their sign with respect the previous cases, in particular Equations (1) and (2), for convenience). The solution is

$$P_j=exp\left(-1+\sum _{i=1}^m{\lambda }_i{f}_i\left(j\right)\right)$$

(4)

$$=exp\left(-1+\sum \lambda {}^{\prime }s \mathrm{of} \mathrm{features} \mathrm{of} \mathrm{word} j\right).$$

In contrast with the previous simplistic models, we are now able to deal with the inner structure of words, as composed by letters, i.e., $j=\{{\ell }_1,{\ell }_2,\dots \}$ and $P_j=P({\ell }_1{\ell }_2\dots ),$ with ${\ell }_1$ the letter at first position of word j and so on. If we consider that the features describe the individual letters of a word, for example, for $i=1$c, then Equation (3) writes

$$\langle f_{1\mathtt{c}}\rangle =\sum _{j=1}^V{P}_j{f}_{1\mathtt{c}}\left(j\right)=\sum _{{\ell }_2=\mathtt{a}}^{\mathtt{z}}\sum _{{\ell }_3=\mathtt{a}}^{\mathtt{z}}\cdots P(\mathtt{c}{\ell }_2{\ell }_3\dots )=P_1^I\left(\mathtt{c}\right)={\rho }_1\left(\mathtt{c}\right)$$

(5)

(using that only words starting with ${\ell }_1=\mathtt{c}$ contribute to the sum); in words, we obtain that the expected value of the feature 1c is the marginal probability $P_1^I\left(\mathtt{c}\right)$ that the first letter in a word is c, which we make equal to its empirical value ${\rho }_1\left(\mathtt{c}\right)$ (which is just the number of tokens with letter c in position 1 divided by the total number of tokens). Notice that we do not impose normalization constrain for the $P_j$’s, as this is implicit in the marginals.

Coming back to the expression for the probabilities, Equation (4), we have, for a three-letter example,

$$P^I\left(\mathtt{cat}\right)=exp({\lambda }_{1\mathtt{c}}+{\lambda }_{2\mathtt{a}}+{\lambda }_{3\mathtt{t}}-1),$$

the label I standing for the fact that the solution is obtained from the constrains of one-letter marginals. Substituting this into the constrain, Equation (5), we arrive to ${\rho }_1\left(\mathtt{c}\right)\propto e^{{\lambda }_{1\mathtt{c}}}$, from where we can take solutions of the form $e^{{\lambda }_{1\mathtt{c}}-1/3}={\rho }_1\left(\mathtt{c}\right)$ and so,

$$P^I\left(\mathtt{cat}\right)={\rho }_1\left(\mathtt{c}\right){\rho }_2\left(\mathtt{a}\right){\rho }_3\left(\mathtt{t}\right)$$

(note that other solutions for the ${\lambda }_{1\mathtt{c}}$’s are possible, but they lead to the same $P^I$’s; in particular, the origin of each potential is not fixed and one could replace, for instance, ${\lambda }_{1{\ell }_1}\to {\lambda }_{1{\ell }_1}+C_1$ for all ${\ell }_1$, provided that the other potentials are modified accordingly to yield the same value of the sum).

This model based on univariate (single-letter) marginals is very simple indeed, and closely related to monkey-typing models [28,29], as we obtain that each word is an independent combination of letters, with each letter having its own probability of occurrence (but depending on its position in the word). In other words, one could think of a monkey typing on a keyboard at random, with each letter having a different probability. However, in addition, the probabilities of each letter change depending if the letter is the first of a word, or the second, etc. (the blank is considered to be special letter, which signals the end of a word). Please note that, although the “classical” extension of these simple monkey-typing models is towards Markov models [47], Stephens and Bialek’s approach [55] takes a different direction.

2.2. Pairwise Constrains

The approach of Stephens and Bialek uses the generalization of the previous model to two-letter features, which leads to constrains over the two-letter marginals. For instance, if the feature $i=12$ca denotes that the word has letter c in position 1 and letter a in 2, then, Equation (3) writes

$$\langle f_{12\mathtt{ca}}\rangle =\sum _{\forall j}{P}_j{f}_{12\mathtt{ca}}\left(j\right)=\sum _{{\ell }_3=\mathtt{a}}^{\mathtt{z}}\sum _{{\ell }_4=\mathtt{a}}^{\mathtt{z}}\cdots P(\mathtt{ca}{\ell }_3\dots )=P_{12}^{II}\left(\mathtt{ca}\right)={\rho }_{12}\left(\mathtt{ca}\right),$$

(6)

with ${\rho }_{12}\left(\mathtt{ca}\right)$ the two-letter marginal, provided by the empirical data,

$${\rho }_{12}\left(\mathtt{ca}\right)=\frac{\mathrm{number} \mathrm{of} \mathrm{tokens} \mathrm{with} \mathtt{c}\mathrm{in} 1\mathrm{and} \mathtt{a} \mathrm{in} 2}{\mathrm{total} \mathrm{number} \mathrm{of} \mathrm{tokens}}.$$

The solution (4), restricted for the particular example of a three-letter word can be written as

$$P^{II}\left(\mathtt{cat}\right)=exp({\lambda }_{12}\left(\mathtt{ca}\right)+{\lambda }_{13}\left(\mathtt{ct}\right)+{\lambda }_{23}\left(\mathtt{at}\right)-1),$$

(7)

using the notation ${\lambda }_{12\mathtt{ca}}={\lambda }_{12}\left(\mathtt{ca}\right)$ for the multipliers, and the label $II$ denoting that we are dealing with theoretical probabilities arising from two-letter features, i.e., two-letter marginals. The same result writes, in general,

$$P^{II}\left({\ell }_1{\ell }_2\dots {\ell }_K\right)=exp\left(-1+\sum _{k=1}^{K-1}\sum _{k^{\prime }=k+1}^K{\lambda }_{k{k}^{\prime }}\left({\ell }_k{\ell }_{k^{\prime }}\right)\right),$$

(8)

with K the word length (in number of letters). Comparing to Boltzmann distribution, as in Equation (2), we can identify the argument of the exponential with the energy (in units of ${\beta }^{-1}$ and with a minus sign) and the Lagrange multiplier for each feature with the pairwise interaction potential between the letters defining the feature (with a minus sign, and with a shift of one unit); for example,

$$-\beta E\left(\mathtt{cat}\right)={\lambda }_{12}\left(\mathtt{ca}\right)+{\lambda }_{13}\left(\mathtt{ct}\right)+{\lambda }_{23}\left(\mathtt{at}\right)-1,$$

and in general,

$$-\beta E\left({\ell }_1{\ell }_2\dots {\ell }_K\right)=-1+\sum _{k=1}^{K-1}\sum _{k^{\prime }=k+1}^K{\lambda }_{k{k}^{\prime }}\left({\ell }_k{\ell }_{k^{\prime }}\right).$$

Therefore, words can be seen as networks of interacting letters (with all-to-all interaction between pairs, and where the position of the two letters matters for the interaction). Please note that three-letter interactions, common in English orthographic rules, are not captured by the pairwise interaction; for example, in positions 3 to 5: believe (rule) versus deceive (exception, due to the c letter). Remarkably, this pairwise approach has been used also for neuronal, biochemical, and genetic networks [55]. A very simplified case of this letter system turns out to be equivalent to an Ising model (or, more properly, a spin-glass model): just consider an alphabet of two letters (a and b) and impose the symmetries (not present in linguistic data, in general) ${\lambda }_{k{k}^{\prime }}\left(\mathtt{a}\mathtt{b}\right)={\lambda }_{k{k}^{\prime }}\left(\mathtt{b}\mathtt{a}\right)$ and ${\lambda }_{k{k}^{\prime }}\left(\mathtt{a}\mathtt{a}\right)={\lambda }_{k{k}^{\prime }}\left(\mathtt{b}\mathtt{b}\right)$ (if one wants to get rid of this symmetry in the Ising system one could consider external “magnetic” fields, associated with the one-letter marginals).

Substituting the solution (4) or (7) into the constrains (6), the equations we need to solve would be like

$$P_{12}^{II}\left(\mathtt{ca}\right)=\langle f_{12\mathtt{ca}}\rangle =\sum _j{f}_{12\mathtt{ca}}\left(j\right)e^{-1+{\sum }_{i=1}^m{\lambda }_i{f}_i\left(j\right)}=$$

$$=e^{{\lambda }_{12}\left(\mathtt{ca}\right)}\sum _{{\ell }_3=\mathtt{a}}^{\mathtt{z}}{e}^{{\lambda }_{13}\left(\mathtt{c}{\ell }_{\mathtt{3}}\right)+{\lambda }_{23}\left(\mathtt{a}{\ell }_{\mathtt{3}}\right)-1}={\rho }_{12}\left(\mathtt{ca}\right),$$

(9)

if we restricted to three-letter words.

For computational limitations, we will only treat words comprising from 1 to 6 letters. As the numerical algorithm we will use requires that the number of letters is constant (see the Appendix A), we will consider that words shorter than length 6 are six-letter words whose last positions are filled with blanks; for example, cat = cat$\square \square \square$, where the symbol □ denotes a blank. In this way, instead of the usual 26 letters in English we deal with 27 (the last term in the sums of some of the previous equations should be □, instead of z). This yields $6\times 5/2=15$ interaction potentials (15 features) for each word, and a total of $15\times {27}^2=\mathrm{10,935}$ unknown values of the interaction potential (i.e., Lagrange multipliers with minus sign) corresponding to 10,935 equations (one for each value of the two-letter marginals). In contrast, note that there are about ${27}^6=\mathrm{387,420,489}$ possible words of length between 1 and 6 (the figures turn out to be a bit smaller if one recalls that blanks can only be at the end of the word, in fact, $26+\dots +{26}^6=\mathrm{321,272,406}$). In more generality, the 10,935 equations to solve are like

$$e^{{\lambda }_{12}\left(\mathtt{ca}\right)}\sum _{{\ell }_3\dots {\ell }_6}{e}^{{\lambda }_{13}\left(\mathtt{c}{\ell }_{\mathtt{3}}\right)+\cdots +{\lambda }_{16}\left(\mathtt{c}{\ell }_{\mathtt{6}}\right)+{\lambda }_{23}\left(\mathtt{a}{\ell }_{\mathtt{3}}\right)+\cdots +{\lambda }_{26}\left(\mathtt{a}{\ell }_{\mathtt{6}}\right)+{\lambda }_{34}\left({\ell }_{\mathtt{3}}{\ell }_{\mathtt{4}}\right)+\dots \cdots +{\lambda }_{56}\left({\ell }_{\mathtt{5}}{\ell }_{\mathtt{6}}\right)-1}={\rho }_{12}\left(\mathtt{ca}\right),$$

(10)

where the solution is not straightforward anymore, and has to be found numerically. Therefore, we deal with a constrained optimization problem, for which the Appendix A provides complete information. Here we just mention that the so-called improved iterative-scaling method [61,62] consist of the successive application of transformations as

$${\lambda }_{12}\left(\mathtt{ca}\right)\to {\lambda }_{12}\left(\mathtt{ca}\right)+\frac{1}{15}ln\frac{{\rho }_{12}\left(\mathtt{ca}\right)}{P_{12}^{II}\left(\mathtt{ca}\right)},$$

see Equation (A1) in the Appendix A, with $P_{12}^{II}\left(\mathtt{ca}\right)$ calculated from the maximum-entropy solution, Equation (9). Please note that, as in the case of univariate marginals, the potentials are undetermined under a shift, i.e., ${\lambda }_{12}\left({\ell }_1{\ell }_2\right)\to {\lambda }_{12}\left({\ell }_1{\ell }_2\right)+C_{12}$, as long as the other potentials are correspondingly shifted to give the same value for the sum.

3. Data and Results

3.1. Data

As a corpus, we use all English books in the recently presented Standardized Project Gutenberg Corpus [63]. This comprises more than 40,000 books in English, with a total number of tokens 2,016,391,406 and a vocabulary size $V=\mathrm{2,268,043}$. The entropy of the corresponding word-probability distribution is $S=10.27$ bits. To avoid spurious words (misspellings, etc.), and also for computational limitations, we disregard word types with absolute frequency smaller than 10,000; the corresponding relative frequencies are below $5\times {10}^{-6}$. Also, word types (unigrams) containing characters different than the plain 26 letters from a to z are disregarded (note that we do not distinguish between capital and lower-case letters). Finally, we remove also Roman numerals (these are not words for our purposes, as they are not formed by interaction between letters). This reduces the number of tokens to 1,881,679,476 and V to 11,042, and so the entropy becomes $S=9.45$ bits. Finally, the subset of words with length smaller or equal to 6 yields 1,597,358,419 tokens, $V=5081$ and $S=8.35$ bits. We will see that these sub-corpora fulfill Zipf’s law, but each one with a slightly different power-law exponent. Please note that the fact of disregarding relative frequencies below $5\times {10}^{-6}$ does not influence the fulfillment of Zipf’s law, as Zipf’s law is a high-frequency phenomenon (see

Table 1. Results of power-law fitting of the form $g\left(\rho \right)\propto 1/{\rho }^{\gamma +1}$ (for $a\le \rho \le b$) applied to the 15 empirical two-letter marginal distributions (with $b=\infty$), to the empirical word frequency ${\rho }_{word}$ and to the theoretical maximum-entropy solution $P^{II}$. The empirical distribution for words of any length, ${\rho }_{allword}$, is also shown, in order to compare it with ${\rho }_{word}$. V is the number of types (pairs of letters or words); ${\rho }_{max}$ is the highest empirical frequency; $j_{max}$ is the corresponding type (pair of letters or word type; the next highest-frequency types appear in brackets); o.m. is the number of orders of magnitude in the fit, ${log}_{10}({\rho }_{max}/a)$; v is the number of types that enter into the power-law fit; $\sigma$ is the standard error of the fitted exponent; and p is the $p–$value of the goodness-of-fit test. The ratio $v/V$ ranges from 0.09 to 0.3. Only words of length from 1 to 6 are taken into account. Blanks are not considered in the marginals. 50 values of a and b (when b is not fixed to ∞) are analyzed per order of magnitude, equally spaced in logarithmic scale. $p–$values are computed from 1000 Monte Carlo simulations. Fits are considered non-rejectable if $p\ge 0.20$.

Distribution	V	${\mathit{\rho }}_{\mathbf{max}}$	${\mathit{j}}_{\mathit{max}}$	a$(\times {\mathbf{10}}^{-\mathbf{4}})$	b	o.m.	v	$\mathit{\gamma }\pm \mathit{\sigma }$	p
${\rho }_{12}$	223	0.143	th (an, of, to, he)	63.1	∞	1.36	40	1.282 ± 0.213	0.21
${\rho }_{13}$	471	0.146	te (i□, o□, a□, ad)	16.6	∞	1.94	133	1.138 ± 0.097	0.24
${\rho }_{14}$	455	0.038	t□ (a□, o□, i□, h□)	34.7	∞	1.04	81	1.391 ± 0.156	0.23
${\rho }_{15}$	391	0.043	t□ (a□, o□, i□, h□)	36.3	∞	1.07	78	1.433 ± 0.175	0.28
${\rho }_{16}$	285	0.042	t□ (a□, o□, w□, i□)	69.2	∞	0.78	45	2.110 ± 0.324	0.23
${\rho }_{23}$	309	0.160	he (f□, o□, □□, nd)	57.5	∞	1.44	42	1.207 ± 0.197	0.29
${\rho }_{24}$	361	0.049	h□ (n□, o□, f□, e□)	60.3	∞	0.91	50	1.466 ± 0.210	0.24
${\rho }_{25}$	334	0.057	h□ (o□, n□, e□, a□)	52.5	∞	1.04	53	1.309 ± 0.183	0.29
${\rho }_{26}$	240	0.055	h□ (o□, n□, e□, a□)	145.0	∞	0.58	21	2.576 ± 0.627	0.22
${\rho }_{34}$	330	0.048	□□ (e□, d□, s□, t□)	83.2	∞	0.76	36	1.764 ± 0.340	0.41
${\rho }_{35}$	371	0.039	□□ (e□, d□, s□, r□)	50.1	∞	0.89	57	1.359 ± 0.190	0.28
${\rho }_{36}$	273	0.045	□□ (e□, d□, r□, t□)	75.9	∞	0.78	44	1.935 ± 0.298	0.32
${\rho }_{45}$	278	0.051	□□ (e□, t□, n□, h□)	87.1	∞	0.77	35	1.579 ± 0.270	0.33
${\rho }_{46}$	244	0.044	□□ (e□, t□, n□, l□)	100.0	∞	0.64	31	1.946 ± 0.378	0.28
${\rho }_{56}$	154	0.115	□□ (e□, s□, d□, t□)	72.4	∞	1.20	34	1.140 ± 0.201	0.58
${\rho }_{allword}$	11042	0.071	the (of, and, to, a)	1.0	0.073	2.85	925	0.925 ± 0.030	0.25
${\rho }_{word}$	5081	0.084	the (of, and, to, a)	0.5	0.087	3.20	1426	0.811 ± 0.023	0.31
$P^{II}$	2174013	0.081	the (of, and, to, a)	0.2	0.083	3.53	2947	0.886 ± 0.017	0.38

).

3.2. Marginal Distributions

Figure 1 displays the empirical two-letter marginal probabilities (obtained from the 6-or-less-letter sub-corpus just described), which constitute the target of the optimization procedure. There are a total of 5092 non-zero values of the marginals. Notice that, although the two-letter marginals are bivariate probabilities (for example, ${\rho }_{12}\left({\ell }_1{\ell }_2\right)$, see also Figure 1a in [55]), Zipf’s representation allows one to display them as univariated. This is achieved by defining a rank variable, assigning rank $r=1$ to the type with the highest empirical frequency $\rho$ (i.e., the most common type), $r=2$ to the second most common type, and so on (Figure 1(left)). This is called the rank-frequency representation (or, sometimes, distribution of ranks), and constitutes a sort of projection of a bivariate (in this case, or multivariate, in general) distribution into a univariate one; for example, ${\rho }_{12}\left({\ell }_1{\ell }_2\right)$, instead of being represented in terms of the random variables ${\ell }_1$ and ${\ell }_2$, is considered a univariate function or the rank, ${\rho }_{12}\left(r\right)$.

Then, Zipf’s law can be formulated as a power-law relation between $\rho$ and r,

$$\rho \propto \frac{1}{r^{1/\gamma }}$$

(11)

for some range of ranks (typpically the lowest ones, i.e., the highest frequencies), with the exponent ${\gamma }^{-1}$ taking values close to one (the symbol ∝ denotes proportionality). When we calculate and report entropies we use always the rank-frequency representation.

An approximated alternative representation [17,64,65], also used by Zipf [14,24], considers the empirical frequency $\rho$ as a random variable, whose distribution is computed. In terms of the complementary cumulative distribution, $G\left(\rho \right)$, Zipf’s law can be written as

$$G\left(\rho \right)\propto \frac{1}{{\rho }^{\gamma }},$$

(12)

which in terms of the probability density or probability mass function of $\rho$ leads to

$$g\left(\rho \right)\propto \frac{1}{{\rho }^{\gamma +1}},$$

(13)

asymptotically, for large $\rho$ (Figure 1(right)). Both $G\left(\rho \right)$ and $g\left(\rho \right)$ constitute a representation in terms of the distribution of frequencies. For more subtle arguments relating $\rho \left(r\right)$, $G\left(\rho \right)$, and $g\left(\rho \right)$, see [17,64,65].

We can test the applicability of Zipf’s law to our two-letter marginals, in order to evaluate how surprising or unsurprising is the emergence from them of Zipf’s law in the word distribution. Remember that, in the case of marginal distributions, types are pairs of letters. Figure 1(left) shows that, despite the number of data in the marginals is relatively low (a few hundred as shown in Table 1, with a theoretical maximum equal to ${26}^2=676$), the marginal frequencies appear as broadly distributed, varying along 4 orders of magnitude (with the frequency $\rho$ in the range from ${10}^{-5}$ to ${10}^{-1}$). Although the double logarithmic plots do not correspond to straight lines, the high-frequency (low-rank) part of each distribution can be fitted to a power law, for several orders of magnitude ranging from 0.5 to 2 and an exponent $\gamma$ typically between 1 and 2, as it can be seen in Table 1. Thus, the two-letter marginal distributions display a certain Zipfian character (at least considering words of length not larger than 6, in letters), with a short power-law range, in general, and with a somewhat large value of $\gamma$ (remember that $\gamma$ has to be close to one for the fulfillment of Zipf’s law).

Remarkably, Figure 1(right) also shows that all the marginal distributions present a characteristic, roughly the same shape, with the only difference being on the scale parameter of the frequency distribution, which is determined by the mean frequency $\langle {\rho }_{k{k}^{\prime }}\rangle$ (denoted generically in the figure as $\langle {\rho }_{emp}\rangle$). This means, as shown in the figure, that the distribution $g\left({\rho }_{emp}\right)$, when multiplied (rescaled) by $\langle {\rho }_{emp}\rangle$, can be considered, approximately, as a function that only depends of the rescaled frequency, ${\rho }_{emp}/\langle {\rho }_{emp}\rangle$, independently on which potential ${\rho }_{k{k}^{\prime }}$ one is considering. In terms on the distribution of ranks this scaling property translates into the fact that ${\rho }_{emp}/\langle {\rho }_{emp}\rangle$ can be considered a function of only $r/V$.

For the fitting we have used the method proposed in [66,67], based on maximum-likelihood estimation and Kolmogorov-Smirnov goodness-of-fit testing. This method lacks the problems presented in the popular Clauset et al.’s recipe [3,68,69]. The fitting method is applied to $\rho$ as a random variable (instead than applied to r [16]); this choice presents several important advantages, as discussed in [65]. The outcome of the method is a estimated value of the exponent $\gamma$ together with a value of $\rho$, denoted by a, from which the power-law fit, Equations (12) and (13), is non-rejectable (with a p-value larger than 0.20, by prescription). Although other distributions different than the power law can be fitted to the marginal data (e.g., lognormal [67]) our purpose is not to find the best fitting distribution, but just to evaluate how much Zipf’s power law depends on a possible Zipf’s behavior of the marginals.

3.3. Word Distributions

Figure 2 shows that the optimization succeeds in getting values of the theoretical marginal distributions very close to the empirical ones. However, despite the fact the target of the optimization are the marginal distributions (whose empirical values are the input of the procedure), we are interested in the distribution of words, whose empirical value is known but does not enter into the procedure, as this is the quantity we seek to “explain”. Zipf’s rank-frequency representation allows us to display in one dimension the six-dimensional nature (from our point of view) of the word frequencies; for the empirical word frequencies this is shown in Figure 3. We find that the distribution is better fitted in terms of an upper truncated power law [66,71], given, as in Equation (13), by $g\left(\rho \right)\propto 1/{\rho }^{\gamma +1}$ but in a finite range $a\le \rho \le b<\infty$ (the untruncated case would be recovered by taking $b\to \infty$). This corresponds, in the continuum case, to a cumulative distribution $G\left(\rho \right)\propto 1/{\rho }^{\gamma }-1/b^{\gamma }$, and to a rank-frequency relation

$$\rho \propto \frac{1}{{(r+V/b^{\gamma })}^{1/\gamma }},$$

which coincides in its mathematical expression with the so-called Zipf-Mandelbrot distribution (although the continuous fit makes r a continuous variable; remember that V is the number of types). The fitting procedure is essentially the same as the one for the untruncated power law outlined in the previous subsection, with the maximization of the likelihood a bit more involved [66,67].

In Figure 3 we also display the theoretical result, $P^{II}$, Equation (8), arising from the solution of Equation (10). We see that, qualitatively, $P^{II}$ has a shape rather similar to the empirical one. Both distributions fulfill Zipf’s law, with exponents $\gamma$ equal to 0.89 and 0.81, respectively. We also see in the figure that the quantitative agreement in the values of the probability ($P^{II}$ and ${\rho }_{word}$) is rather good for the smallest values of the rank ($r<10$); however, both curves start to slightly depart from each other for $r>10$. In addition, the rank values are associated with the same word types for $r\le 6$ (the, of, and, to, a, in), but for larger ranks the correspondence may be different ($r=7$ corresponds to i in one case and to that in the other). If we could represent ${\rho }_{word}$ and $P^{II}$ in six dimensions (instead that as a function of the rank) we would see more clearly the differences between both.

Zipf’s law is, in part, the reason of this problem, as for $r\ge 10$ the difference in probabilities for consecutive ranks becomes smaller than 10 %, see Equation (11), and for $r\ge 100$ the difference decreases to less than 1 % (assuming $\gamma \simeq 1$). Therefore, finite resolution in the calculation of $P^{II}$ will lead to the “mixing of the ranks.” However, the main part of the problem comes from the unability of the algorithm in some cases to yield values of $P^{II}$ close to the empirical value, ${\rho }_{word}$, as it can be seen in the scatter plot of Figure 4 (in agreement with [55]). The entropy of the theoretical word probabilities turns out to be $S=9.90$ bits, somewhat larger than the corresponding empirical value $8.35$ bits. If we truncate this distribution, eliminating probabilities below 10,000/1,597,358,419 $\simeq 6\times {10}^{-6}$ (as in the empirical distribution) we get $S=8.88$ bits, still larger than the empirical value, which simply means that real language has more restrictions than those imposed by the model. Existing (empirical) words for which the algorithm yields the lowest theoretical probabilities are enumerated in the caption of the figure. Curiously, as it can be seen, these are not particularly strange words.

An interesting issue is that the maximum-entropy solution, Equation (8), leads to the “discovery” of new words, or, more properly, pseudowords. Indeed, whereas the empirical corpus has $V=5081$ (number of word types), the theoretical solution leads to $V=$ 2,174,013 (words plus pseudowords). Most of these pseudowords have very small probabilities; however, there are others far from being rare (theoretically). In this way, the most common pseudoword (theoretical word not present in the empirical corpus) is whe, with a theoretical rank $r=40$ (it should be the 40-th most common word in English, for length six or below, following the maximum-entropy criterion).

Table 2. Most common theoretical words from the maximum-entropy procedure that are not present in the analyzed sub-corpus. In fact, all of these theoretical words are present in the original complete corpus (but not in our sub-corpus as we have disregarded frequencies smaller than 10,000). We can distinguish four different cases: the (theoretical) word does not exist in a dictionary (∄) and can be considered a pseudoword (and appears in the complete corpus probably as a misspelling); the word exists in a dictionary as a word (∃); the word appears in a dictionary as an archaism (arch.); and the word appears as an abbreviation (abbrev.). r is (theoretical) rank and $P^{II}$ is (theoretical) probability.

r	${\mathit{P}}^{\mathit{II}}$	Word	Case
40	2.88 $\times {10}^{-3}$	whe	∄
48	2.20 $\times {10}^{-3}$	wis	abbrev.
52	1.95 $\times {10}^{-3}$	mo	abbrev.
61	1.74 $\times {10}^{-3}$	wast	arch.
64	1.69 $\times {10}^{-3}$	ond	∄
71	1.52 $\times {10}^{-3}$	ar	abbrev.
77	1.40 $\times {10}^{-3}$	ane	∃
87	1.24 $\times {10}^{-3}$	ald	abbrev.
89	1.21 $\times {10}^{-3}$	bo	∃
92	1.16 $\times {10}^{-3}$	thes	∄
94	1.10 $\times {10}^{-3}$	hime	∄
98	9.83 $\times {10}^{-4}$	hive	∃
102	9.45 $\times {10}^{-4}$	thise	∄
103	9.39 $\times {10}^{-4}$	af	abbrev.
110	8.80 $\times {10}^{-4}$	wer	∄
117	8.16 $\times {10}^{-4}$	thay	∄
118	8.16 $\times {10}^{-4}$	hes	∄
123	7.88 $\times {10}^{-4}$	wath	∃
125	7.82 $\times {10}^{-4}$	hor	abbrev.
127	7.60 $\times {10}^{-4}$	sime	∄
134	7.22 $\times {10}^{-4}$	tome	∃
135	7.21 $\times {10}^{-4}$	har	∃
141	6.94 $\times {10}^{-4}$	thit	∄
143	6.86 $\times {10}^{-4}$	mas	abbrev.
146	6.77 $\times {10}^{-4}$	hew	∃

provides the first 25 of these pseudowords, ranked by their theoretical probability $P^{II}$. We see that the orthography of these pseudowords looks very “reasonable” (they look like true English words). On the other side, the most rare pseudowords, with probability $P^{II}\sim {10}^{-30}$, are nearly impossible English words, as: sntnut, ouoeil, oeoeil, sntnu, snsnua… (not in the table).

3.4. Values of Lagrange Multipliers and Potentials

We have established that, for a given word, the value of its occurrence probability $P^{II}$ comes from the exponentiation of the sum the 15 interaction potentials between the six letter positions that constitute the word (in our maximum-entropy approach). Therefore, the values of the potentials (or the values of the Lagrange multipliers) determine the value of the probability $P^{II}$. It is interesting to investigate, given a potential or a multiplier (for instance ${\lambda }_{12}$), how the different values it takes (${\lambda }_{12}\left(\mathtt{a}\mathtt{a}\right),{\lambda }_{12}\left(\mathtt{a}\mathtt{b}\right)$, etc.) are distributed. Curiously, we find that the 15 different potentials are (more or less) equally distributed, i.e., follow the same skewed and spiky distribution, as shown in Figure 5(left).

One can try to use this fact to shed some light on the origin of Zipf’s law. Indeed, exponentiation is a mechanism of power-law generation [44,68]. We may arguee that the sum of 15 random numbers drawn from the same spiky distribution has to approach, by the central limit theorem, a normal distribution, and therefore, the exponentiation of the sum would yield a lognormal distribution for $P^{II}$ (i.e., a lognormal shape for $g\left(P^{II}\right)$). However, this may be true for the central part of the distribution, but not for its rightmost extreme values, which is the part of the distribution we are more interested in (high values of $P^{II}$, i.e., the most common words). Note also that, in practice, for calculating the probability of a word, we are not summing 15 equally distributed independent random numbers, as not all the words are possible; i.e., there are potentials that take a value equal to infinite, due to forbidden combinations, and these infinite values are not taken into account in the distribution of the potentials. An additional problem with this approach is that, although most values of the potentials converge to a fix value (and the distribution of potentials shown in the figure is stable), there are single values that do not converge, related to words with very low probability. These issues need to be further investigated in future research. In addition, Figure 5(right) shows, as a scatter plot, the dependence between the value of each potential and the corresponding two-letter marginal probability. Although Equation (10) seems to indicate a rough proportionality between both, the figure shows that such proportionality does not hold (naturally, the rest of terms in the equation play their role).

4. Discussion

We have generalized a previous study of Stephens and Bialek [55]. Instead of restricting our study to four-letter words, we consider words of any length from one to six, which leads to greater computational difficulties, and employ a much larger English corpus as well. We perform an analysis of the fulfillment of Zipf’s law using state-of-art statistical tools. Our more general results are nevertheless in the line of those of [55]. We see how the frequency of occurrence of pairs of letters in words (the pairwise marginal distributions), together with the maximum-entropy principle (which provides the distribution with the maximum possible randomness), constrain the probabilities of word occurrences in English.

Regarding the shape of the distributions, the agreement between the maximum-entropy solution for the word distribution and its empirical counterpart is very good at the qualitative level, and reasonably good at the quantitative level for the most common words, as shown in Figure 3. Moreover, new possible English words, or pseudowords, not present in the corpus (or, more exactly, in the subcorpus we have extracted) have been “discovered”, with hypothetical (theoretical) values of the occurrence probability that vary along many orders of magnitude. However, regarding the probabilities of concrete words, the method yields considerable scatter of the theoretical probabilities (in comparison with the known empirical probabilities), except for the most common words, see Figure 4.

As two by-products, we have found that the pairwise (two-letter) occurrence distributions are all characterized by a well defined shape, see Figure 1(right), and that the distributions of the 15 different interaction potentials are nearly the same, see Figure 5(left). The latter is an intriguing result for two reasons. First, the fact that the values that the 15 interaction potentials take are more or less the same for all of them (Figure 5(left)) seems to indicate that all the potentials are equally important; nevertheless, remember the potentials are undefined with respect one additive constant, and comparison of their absolute values is misleading. Second, not only the values that the potentials take are nearly the same, but they seem to be equally distributed. We have tried to relate, without success yet, this distribution to other skewed and spiky distributions that appear in complex and correlated systems, such as the so-called Bramwell-Holdsworth-Pinton (BHP) distribution [72], the Tracy-Widom distribution, or the Kolmogorov-Smirnov distribution [73,74].

Despite our results, one could still abandon the all-to-all interaction and embrace instead nearest-neighbor coupling (this may seem similar to a Markov model, however, the study of the possible similarities or not should be the subject of a future work). Nearest-neighbor coupling reduces the number of potentials from 15 to 5 (with open boundary conditions), with the subsequent computational simplification. A further reduction would be to impose that all potentials are the same (i.e., they do not depend on letter positions, only on difference of positions, e.g., ${\lambda }_{12}={\lambda }_{23}$, etc.). This leads to only one potential (in the case of nearest-neighbor interaction; 5 potentials in the all-to-all case). It would be interesting to compare these modifications with the original model and to confirm that they lead to much worse results; this is left for future research.

An extension towards a different direction would be to use phonemes or syllables instead of letters as the constituents of words. We urge the authors of the corpus in [63] to provide the decomposition of the words in the corpus into these parts. Naturally, other languages than English should be studied as well. An interesting issue is if our approach (which is that of [55]) can shed light on other linguistic laws; in particular, the word-length law [75,76] and the brevity law (also called Zipf’s law of abbreviation [75,76,77]). Therefore, we could verify up to which point longer words have smaller theoretical probabilities, and if the robust patterns found in [76] are also valid for the maximum-entropy solution. Furthermore, we could quantify the role of the pairwise interaction potentials in determining the length of the word (the “brevity”), looking at the interaction of any of the 26 letters with the blank. Alternatively, one could study the word frequency distributions at fixed length [76], and check if the potentials are stable for different word lengths. Finally, let us mention that the approach presented here has also been applied to music [78]. This, together with the applications in neuronal, biochemical, and genetic networks mentioned above ([55] and references therein) confirms the high interdisciplinarity of this approach.