Efficient First-Order Algorithms for Large-Scale, Non-Smooth Maximum Entropy Models with Application to Wildfire Science

Abstract

Maximum entropy (MaxEnt) models are a class of statistical models that use the maximum entropy principle to estimate probability distributions from data. Due to the size of modern data sets, MaxEnt models need efficient optimization algorithms to scale well for big data applications. State-of-the-art algorithms for MaxEnt models, however, were not originally designed to handle big data sets; these algorithms either rely on technical devices that may yield unreliable numerical results, scale poorly, or require smoothness assumptions that many practical MaxEnt models lack. In this paper, we present novel optimization algorithms that overcome the shortcomings of state-of-the-art algorithms for training large-scale, non-smooth MaxEnt models. Our proposed first-order algorithms leverage the Kullback–Leibler divergence to train large-scale and non-smooth MaxEnt models efficiently. For MaxEnt models with discrete probability distribution of n elements built from samples, each containing m features, the stepsize parameter estimation and iterations in our algorithms scale on the order of $O\left(mn\right)$ operations and can be trivially parallelized. Moreover, the strong ${\ell }_1$ convexity of the Kullback–Leibler divergence allows for larger stepsize parameters, thereby speeding up the convergence rate of our algorithms. To illustrate the efficiency of our novel algorithms, we consider the problem of estimating probabilities of fire occurrences as a function of ecological features in the Western US MTBS-Interagency wildfire data set. Our numerical results show that our algorithms outperform the state of the art by one order of magnitude and yield results that agree with physical models of wildfire occurrence and previous statistical analyses of wildfire drivers.

1. Introduction

Maximum entropy (MaxEnt) models are a class of density estimation methods that use the maximum entropy principle [1] to reproduce key statistics of data sets. Historically used in physics [1,2], engineering [3,4,5], and statistics [6] applications, MaxEnt models are now frequently used for large-scale machine learning problems in natural language processing [7,8,9,10,11,12], social science [13,14,15], neuroscience [16,17,18], ecological modeling [19,20,21,22,23,24,25,26,27,28], and wildfire science [29,30,31]. MaxEnt models in machine learning must often estimate probability distributions from big data sets comprising hundreds of thousands to billions of samples or features or both [32]. Due to this enormous number, large-scale MaxEnt models need efficient and robust algorithms to perform well.

State-of-the-art algorithms for MaxEnt models, however, were not originally designed to handle big data sets; these algorithms either rely on technical devices that may yield unreliable numerical results [33], scale poorly in size [9,34], or require smoothness assumptions that many MaxEnt models lack in practice [10]. These limitations make it essentially impossible to scale MaxEnt models to big data applications without adequate and costly computational resources [35,36]. This constraint on computational resources, in particular, has been recently identified as a crucial challenge to overcome for using large-scale MaxEnt models in climate change and wildfire studies within a reasonable amount of run time [27,28,30,37].

1.1. Contributions of This Paper

In this paper, we present novel optimization algorithms that overcome the shortcomings of state-of-the-art algorithms used for training large-scale, non-smooth MaxEnt models. Our proposed algorithms are first-order accelerated nonlinear primal–dual hybrid gradient (NPDHG) algorithms, whose theory was provided by two authors of this paper in [38], based on the Kullback–Leibler divergence. Using the Kullback–Leibler divergence over the classical proximal operator makes it possible to train large-scale and non-smooth MaxEnt models much more efficiently than the state of the art. In particular, we show that, for a MaxEnt model with a discrete probability distribution of n elements built from samples each containing m features, the stepsize parameter estimation and iterations in our algorithms all scale on the order of $O\left(mn\right)$ operations. This significantly improves on the known complexity bound of $O\left(\min (m^{2}n,m{n}^2)\right)$ operations for computing the optimal stepsize parameters of classical first-order optimization methods, such as the linear PDHG or forward–backward splitting methods. We also show, as a consequence, that, for a given tolerance level $ϵ>0$, our algorithms provably compute solutions using on the order of $O(mn/\sqrt{ϵ})$ or $O(mn/\log (1/ϵ\left)\right)$ operations, the order depending on the smoothness of the MaxEnt model and which are optimal with respect to the Nesterov class of optimal first-order methods [39]. Moreover, the computational bottleneck consists of matrix–vector multiplications, which can be trivially parallelized, and so our proposed algorithms exhibit scalable parallelism. Finally, we show that the strong convexity of the Kullback–Leibler divergence with respect to the ${\ell }_1$ norm naturally allows for significantly larger stepsize parameters, thereby substantially speeding up the convergence rate of our algorithms.

To illustrate the efficiency of our novel algorithms on large-scale problems, we present an application to wildfire science. Specifically, we consider the problem of estimating the probabilities of fire occurrences across the western US as a function of ecological features. To do so, we fit elastic net, non-overlapping group lasso, and ${\ell }_{\infty }$ MaxEnt models to a large number of hyperparameters using our proposed algorithms and the state-of-the-art forward–backward splitting and coordinate descent STRUCTMAXENT2 algorithms [40,41]. Our numerical results show that our algorithms outperform the two latter algorithms by at least one order of magnitude, and yield results that are in good agreement with physical models of wildfire occurrence [42,43] as well as previous statistical analyses of wildfire drivers [29,30,31,44].

1.2. Organization of This Paper

This paper is organized as follows. In Section 2, we present our setup, we describe how MaxEnt models work, and we review three popular non-smooth MaxEnt models that are challenging to train with big data sets: the elastic net, group lasso, and ${\ell }_{\infty }$ regularized MaxEnt models. In Section 3, we explain why large-scale, non-smooth MaxEnt models are particularly challenging to train from big data sets, and we describe the state-of-the-art algorithms for training these MaxEnt models and their limitations. In Section 4, we describe our approach for estimating MaxEnt models with NPDHG optimization methods, derive explicit algorithms for regularized MaxEnt models, and explain why our algorithms overcome the limitations of state-of-the-art algorithms. In Section 5, we present an application of our algorithms to wildfire science, where we train a large number of MaxEnt models on the Western US MTBS-Interagency (WUMI) wildfire data set [45] to estimate the probabilities of fire occurrences as a function of ecological features. Finally, we review our results and outline directions for future work in Section 6.

2. Preliminaries

2.1. Setup

Suppose we receive l independent, identically distributed samples $\{v_1,\dots ,v_l\}\subset \mathcal{I}$ from an unknown distribution $\mathcal{D}$. We assume throughout, without of loss of generality, that the input space $\mathcal{I}\equiv \{1,\dots ,n\}$. In addition, suppose we have a prior probability distribution $p_{\mathrm{prior}}$ on $\mathcal{I}$ that encapsulates some prior knowledge about the samples or unknown distribution. Finally, suppose we have access to a set of features from the samples via a bounded feature map $\mathbf{\Phi }:\mathcal{I}\to {\mathbb{R}}^m$. Then, how do we estimate the unknown distribution $\mathcal{D}$ from the prior distribution $p_{\mathrm{prior}}$, the samples $\{v_1,\dots ,v_l\}$, and the feature map $\mathbf{\Phi }$?

The maximum entropy principle offers a systematic way to answer this question. It states that the distribution that best estimates the unknown distribution $\mathcal{D}$ is the one that remains as close as possible to the prior probability $p_{\mathrm{prior}}$ while matching the features $\{\mathbf{\Phi }\left(v_1\right),\dots ,\mathbf{\Phi }\left(v_l\right)\}$ exactly or as closely as possible, in some suitable sense. Specifically, let ${\Delta }_n$ denote the n-dimensional probability simplex and suppose $p_{\mathrm{prior}}\in \mathrm{int}{\Delta }_n$ (i.e., $p_{\mathrm{prior}}\left(j\right)>0$ for every $j\in \{1,\dots ,n\}$). We measure the closeness of a probability distribution $p\in {\Delta }_n$ to the prior probability $p_{\mathrm{prior}}$ using the Kullback–Leibler divergence $D_{\mathrm{KL}}:{\Delta }_n\times \mathrm{int}{\Delta }_n\to \mathbb{R}$:

$$D_{\mathrm{KL}}(p\parallel p_{\mathrm{prior}})=\sum _{j=1}^{n}p\left(j\right)\log \left(p\left(j\right)/p_{\mathrm{prior}}\left(j\right)\right).$$

(1)

Next, let $\widehat{\mathcal{D}}$ denote the empirical distribution induced by samples $\{v_1,\dots ,v_l\}$:

$$\widehat{\mathcal{D}}\left(j\right)={\displaystyle \frac{1}{l}}\left|\{1⩽i⩽l\mid v_i=j\}\right|.$$

(2)

In addition, let ${\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]$ denote the empirical average of the features induced by the samples and let ${\mathbb{E}}_p[\mathbf{\Phi }]$ denote the model average induced by a probability distribution $p\in {\Delta }_n$:

$${\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]=\sum _{j=1}^n\widehat{\mathcal{D}}\left(j\right)\mathbf{\Phi }\left(j\right)\mathrm{and}{\mathbb{E}}_p[\mathbf{\Phi }]=\sum _{j=1}^{n}p\left(j\right)\mathbf{\Phi }\left(j\right).$$

We measure how the averages ${\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]$ and ${\mathbb{E}}_p[\mathbf{\Phi }]$ match using a function $H^{*}:{\mathbb{R}}^m\to \mathbb{R}\cup \{+\infty \}$ called the potential function. MaxEnt models combine the two measures described above and seek to minimize the sum

$$\underset{p\in {\Delta }_n}{\inf }f(p;t)=\underset{p\in {\Delta }_n}{inf}\left\{D_{\mathrm{KL}}(p\parallel p_{\mathrm{prior}})+t{H}^{*}\left({\displaystyle \frac{{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_p[\mathbf{\Phi }]}{t}}\right)\right\}$$

(3)

where $t>0$ is a hyperparameter selected using data-driven methods, e.g., cross-validation.

Standing assumptions: The potential function is proper, lower semicontinuous, convex, and bounded from below by zero with $H^{*}\left(\mathbf{0}\right)$ = 0. Under the standing assumptions above on the potential function, which are fairly general, the probability distribution $\widehat{\mathcal{D}}$ is a feasible point of (3). Moreover, the MaxEnt estimation problem (3) admits a unique global solution (see ([46], Page 35, Proposition 1.2); uniqueness follows from the strong convexity of the Kullback–Leibler divergence with respect to the ${\ell }_1$ norm [47,48,49,50,51]).

2.2. Dual Formulation and Optimality Conditions

The generalized MaxEnt problem (3) admits a dual problem corresponding to regularized maximum a posteriori estimation [52]. To derive the dual problem, we first write the second term on the right hand side of (3) in terms of its convex conjugate,

$$t{H}^{*}\left({\displaystyle \frac{{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_p[\mathbf{\Phi }]}{t}}\right)=\underset{\mathit{w}\in {\mathbb{R}}^m}{sup}\left\{〈\mathit{w},{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_p[\mathbf{\Phi }]〉-tH\left(\mathit{w}\right)\right\},$$

where we abuse notation and write the convex conjugate of $H^{*}$ as H. This lets us express problem (3) in saddle-point form:

$$\underset{p\in {\Delta }_n}{inf}\underset{\mathit{w}\in {\mathbb{R}}^m}{sup}\left\{〈\mathit{w},{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_p[\mathbf{\Phi }]〉-tH\left(\mathit{w}\right)+D_{\mathrm{KL}}(p\parallel p_{\mathrm{prior}})\right\}.$$

(4)

Due to our assumptions on the potential function $H^{*}$, we can swap the infimum and supremum ([46], Statement (4.1) on page 61). Using the convex conjugate formula

$$\underset{p\in {\Delta }_n}{inf}\left\{D_{\mathrm{KL}}(p\parallel p_{\mathrm{prior}})-〈\mathit{w},{\mathbb{E}}_p[\mathbf{\Phi }]〉\right\}=-\log \left[\sum _{j=1}^n{p}_{\mathrm{prior}}\left(j\right)e^{〈\mathit{w},\mathbf{\Phi }\left(j\right)〉}\right],$$

we obtain the dual problem of (3)

$$\underset{\mathit{w}\in {\mathbb{R}}^m}{sup}\left\{〈\mathit{w},{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]〉-tH\left(\mathit{w}\right)-\log \left[\sum _{j=1}^n{p}_{\mathrm{prior}}\left(j\right)e^{〈\mathit{w},\mathbf{\Phi }\left(j\right)〉}\right]\right\}.$$

(5)

The dual problem (5) is a regularized maximum likelihood estimation problem over the family of Gibbs distributions [41,52]. It has at least one global solution ([46], Proposition (1.2) on page 35). Moreover, if $(p^s,{\mathit{w}}^s)$ denotes a pair of solutions to the MaxEnt estimation problem (3) and its dual (5), then this pair satisfies the optimality conditions

$${\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p^s}[\mathbf{\Phi }]\in t\partial H\left({\mathit{w}}^s\right)\mathrm{and}{p}^s\left(j\right)={\displaystyle \frac{p_{\mathrm{prior}}\left(j\right)e^{\langle {\mathit{w}}^s,\mathbf{\Phi }\left(j\right)\rangle }}{{\sum }_{j=1}^n{p}_{\mathrm{prior}}\left(j\right)e^{\langle {\mathit{w}}^s,\mathbf{\Phi }\left(j\right)\rangle }}},$$

(6)

for every $j\in \{1,\dots ,n\}$ where $\partial H\left({\mathit{w}}^s\right)$ denotes the subdifferential of H at ${\mathit{w}}^s$.

2.3. Examples of Non-Smooth MaxEnt Models

MaxEnt models differ in the choice of the prior distribution $p_{\mathrm{prior}}$, hyperparameter t, and the potential function $H^{*}$. In practice, the prior distribution is often chosen to be uniform and the hyperparameter is chosen via data-driven methods. The choice of potential function depends on the application. The classical MaxEnt model uses the indicator function

$$\mathit{u}\mapsto H^{*}\left(\mathit{u}\right)=\{\begin{array}{cc}0,\hfill & \mathrm{if}\mathit{u}=\mathbf{0},\hfill \\ +\infty \hfill & \mathrm{otherwise}.\hfill \end{array}$$

This forces the model average ${\mathbb{E}}_p[\mathbf{\Phi }]$ to be equal to the empirical average ${\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]$. However, this is often too restrictive because the averages are expected to be close, and not equal, with high probability. Forcing the model and empirical averages to be equal can lead to severe over-fitting issues with big data sets, and more flexibility is therefore desired.

We review below three different non-smooth MaxEnt models that offer this flexibility: the elastic net, group lasso, and ${\ell }_{\infty }$-regularized MaxEnt models. We chose these models because they are used extensively in statistics and machine learning but are also challenging to train on big data sets due to their non-smoothness [20,26,27,53]. These models will also be used for our numerical experiments and results in Section 5.

2.3.1. Elastic Net Regularized MaxEnt Models

These models use for potential function the relaxation of the convex conjugate of the ${\ell }_1$-norm:

$$\mathit{u}\mapsto H^{*}\left(\mathit{u}\right)\equiv {\left({\displaystyle \frac{1-\alpha }{2}}{∥·∥}_2^2+\alpha {∥·∥}_1\right)}^{*}\left(\mathit{u}\right)={\displaystyle \frac{1}{2(1-\alpha )}}\sum _{i=1}^{m}max(0,|u_i{|-\alpha )}^2$$

where $\alpha \in (0,1)$. This yields the elastic net regularized maximum entropy estimation problem

$$\underset{p\in {\Delta }_n}{\min }\left\{D_{\mathrm{KL}}p(p\parallel p_{\mathrm{prior}})+{\displaystyle \frac{1}{2t(1-\alpha )}}\sum _{i=1}^{m}max{\left(0,\left|{\left[{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]\right]}_i-{\left[{\mathbb{E}}_p[\mathbf{\Phi }]\right]}_i\right|-t\alpha \right)}^2\right\}.$$

(7)

This type of regularization is frequently used for feature selection in machine learning [12,20,54,55]. The corresponding dual problem is

$$\underset{\mathit{w}\in {\mathbb{R}}^m}{sup}\left\{〈\mathit{w},{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]〉-t\left({\displaystyle \frac{(1-\alpha )}{2}}{∥\mathit{w}∥}_2^2+\alpha {∥\mathit{w}∥}_1\right)-\log \left[\sum _{j=1}^n{p}_{\mathrm{prior}}\left(j\right)e^{〈\mathit{w},\mathbf{\Phi }\left(j\right)〉}\right]\right\}.$$

(8)

Thanks to the elastic net penalty ${\displaystyle \frac{(1-\alpha )}{2}}{∥\mathit{w}∥}_2^2+\alpha {∥\mathit{w}∥}_1$, problem (8) is strongly concave and has a unique solution.

An important aspect of the elastic net penalty is that it promotes sparsity; that is, it leads to a solution with many entries equal to zero [56,57,58], the number depending on the hyperparameter t and parameter $\alpha$. Sparsity is useful when seeking accurate solutions based only on a few features or when performing feature selection.

2.3.2. Group Lasso Regularized MaxEnt Models

These models are specified starting from the dual problem. Consider a partition G of the components of a vector $\mathit{w}\in {\mathbb{R}}^m$ into g different and possibly overlapping groups $[{\mathit{w}}_1,\dots ,{\mathit{w}}_g]$ with ${\mathit{w}}_g\in {\mathbb{R}}^{m_g}$ and ${\cup }_{g=1}^G{\mathit{w}}_g=\mathit{w}$. Let $\mathit{w}\mapsto {∥{\mathit{w}}_g∥}_{2,g}$ denote the ${\ell }_2$ norm over the respective components of the gth group. Then, the dual version of the group lasso regularized maximum entropy estimation problem is

$$\underset{\mathit{w}\in {\mathbb{R}}^m}{sup}\left\{\langle \mathit{w},{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]\rangle -t\sum _{g=1}^G\sqrt{m_g}{∥{\mathit{w}}_g∥}_{2,g}-\log \left[\sum _{j=1}^n{p}_{\mathrm{prior}}\left(j\right)e^{〈\mathit{w},\mathbf{\Phi }\left(j\right)〉}\right]\right\}.$$

(9)

Thanks to the group lasso penalty ${\displaystyle \sum _{g=1}^G}\sqrt{m_g}{∥{\mathit{w}}_g∥}_{2,g}$, problem (10) is strictly concave and so has at least one global solution. The corresponding primal problem follows from the convex conjugate formula

$$\mathit{u}\mapsto H^{*}\left(\mathit{u}\right)\equiv {\left(\sum _{g=1}^G\sqrt{m_g}{∥·∥}_{2,g}\right)}^{*}\left(\mathit{u}\right)=\{\begin{array}{c}0,\mathrm{if}{∥{\mathit{u}}_g∥}_{2,g}⩽\sqrt{m_g}\mathrm{for}g=1,\dots ,G,\hfill \\ +\infty ,\mathrm{otherwise},\hfill \end{array}$$

and it is therefore given by

$$\underset{p\in {\Delta }_n}{\min }{D}_{\mathrm{KL}}(p\parallel p_{\mathrm{prior}})\mathrm{s}.\mathrm{t}.{∥{\mathbb{E}}_{\widehat{\mathcal{D}}}\left[{\mathbf{\Phi }}_g\right]-{\mathbb{E}}_p\left[{\mathbf{\Phi }}_g\right]∥}_{2,g}⩽t\sqrt{m_g}\mathrm{for}g=1,\dots ,G.$$

(10)

Similarly to the elastic net penalty, the group lasso penalty promotes sparsity in the global solutions of (9), but it differs in that it sets different blocks of components $[{\mathit{w}}_1,\dots ,{\mathit{w}}_g]$ to zero at the same time.

2.3.3. ${\ell }_{\infty }$-Regularized MaxEnt Models

These models use for potential function the convex conjugate of the ${\ell }_{\infty }$ norm,

$$\mathit{u}\mapsto H^{*}\left(\mathit{u}\right)\equiv {∥\mathit{u}∥}_{\infty }^{*}=\left\{\widehat{\mathit{u}}\in {\mathbb{R}}^m\mid {∥\mathit{u}-\widehat{\mathit{u}}∥}_1⩽1\right\},$$

which is the characteristic set of the ${\ell }_1$ ball with unit radius. This yields the ${\ell }_{\infty }$-regularized maximum entropy estimation problem

$$\underset{p\in {\Delta }_n}{\min }\left\{D_{\mathrm{KL}}(p\parallel p_{\mathrm{prior}})+t{∥{\displaystyle \frac{{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_p[\mathbf{\Phi }]}{t}}∥}_{\infty }^{*}\right\}.$$

(11)

This type of regularization is used for certain machine learning problems, including matrix factor decomposition, and in robust statistics applications (see, e.g., [53,59]). The corresponding dual problem is

$$\underset{\mathit{w}\in {\mathbb{R}}^m}{sup}\left\{\langle \mathit{w},{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]\rangle -t{∥\mathit{w}∥}_{\infty }-\log \left[\sum _{j=1}^n{p}_{\mathrm{prior}}\left(j\right)e^{〈\mathit{w},\mathbf{\Phi }\left(j\right)〉}\right]\right\}$$

(12)

Thanks to the ${\ell }_{\infty }$ norm, problem (12) is strictly concave and has at least one global solution.

3. Related Work

3.1. Large-Scale Sparse MaxEnt Models: Computational Challenges

Estimating a probability distribution from the MaxEnt model (3) can be computationally prohibitive for big data sets. To illustrate this point, suppose the MaxEnt model (3) with hyperparameter $t⩾0$ has a global solution $p^s\left(t\right)$. Let $p^{ϵ}\left(t\right)\in {\Delta }_n$ with $ϵ>0$ denote an $ϵ$-approximate solution to the global solution $p^s\left(t\right)$, that is, the objective function f in (3) satisfies

$$f(p^{ϵ}\left(t\right);t)-f(p^s\left(t\right);t)<ϵ.$$

If the potential function $H^{*}$ is smooth (equivalently, its conjugate H is strongly convex), the best achievable rate of convergence for computing $p^{ϵ}\left(t\right)$ with $t>0$ in the Nesterov class of optimal first-order methods is linear $O\left(\log \right(1/ϵ\left)\right)$ in the number of operations [39]. If H is not smooth, then the optimal convergence rate is sublinear $O(1/\sqrt{ϵ})$ in the number of operations.

These rates, while optimal, require carefully fine-tuned stepsize parameters. In classical first-order optimization algorithms, these stepsize parameters are fine-tuned using a precise estimate of the largest singular value of the linear operator $\mathit{A}:{\Delta }_n\to {\mathbb{R}}^m$ defined by

$$\mathit{A}p=\sum _{j=1}^{n}p\left(j\right)\mathbf{\Phi }\left(j\right)={\mathbb{E}}_p[\mathbf{\Phi }].$$

(13)

Unfortunately, computing the largest singular value of the linear operator $\mathit{A}$ in (13) accurately is often computationally expensive for large matrices due to its prohibitive cubic cost of $O\left(\min (m^{2}n,m{n}^2)\right)$ operations [60]. In this situation, line search methods and other heuristics can sometime be employed to bypass this issue, but they typically slow down the convergence speed. Another approach is to compute a crude estimate of the largest singular value of the matrix $\mathit{A}$, but doing so significantly reduces convergence speed as well. In fact, even if the largest singular value can be computed quickly, the resulting stepsize parameters may be much smaller than what is permissible to maintain convergence, that is, the largest singular value itself may be a poor estimate for determining how large the stepsize parameters are allowed to be to maintain convergence. This point will discussed in more detail in Section 4.1 when we describe our proposed NPDHG algorithms.

This issue makes solving the MaxEnt model (3) difficult and laborious. Even worse, in some applications, the appropriate value of the hyperparameter t in (3) is difficult to guess and must be selected by repeatedly solving (3) from a large pool of values of hyperparameters, a process that can become particularly time-consuming and resource intensive for big data sets. This issue has driven much research in developing robust and efficient algorithms to minimize computational costs and maximize model performance.

3.2. State-of-the-Art Methods for Large-Scale, Non-Smooth MaxEnt Models

State-of-the-art methods for computing solutions to large-scale, non-smooth MaxEnt models are based on coordinate descent algorithms [61,62,63,64] and first-order optimization algorithms such as forward–backward splitting [40,65,66]. We will discuss these methods below, but note that other types of methods have been developed for MaxEnt models; see, e.g., [10,41,67,68] for surveys and comparisons of different algorithms. In particular, we will not consider second-order-based methods suitable for smooth MaxEnt models such as limited-memory BFGS algorithms [10,69] since this work focuses on non-smooth MaxEnt models.

Coordinate descent methods. The state of the art is a coordinate descent algorithm based on a technical device called infinitely weighted logistic regression (IWLR) [26,33]. The IWLR method approximates a MaxEnt model as a logistic regression model and then fits the approximate logistic regression model using an existing, efficient optimization algorithm for logistic regression. This indirect approach was proposed because efficient and scalable coordinate descent algorithms were already available for fitting logistic regression models, which in [33] was an earlier version of the GLMNET software package [63]. The IWLR method remains the state of the art for this reason. It is implemented, for example, in the MaxEnt package available in the R programming language [26,63].

The IWLR method, however, is an approximate technical device that is not guaranteed to work, a fact acknowledged by the authors who proposed the method [33], and therefore may not produce reliable numerical results. Coordinate descent algorithms themselves are generally non-parallelizable and often lack robustness and good convergence properties. For example, the aforementioned GLMNET software package approximates the logarithm term in logistic regression models with a quadratic term to fit the models efficiently. Without costly stepsize optimization, which GLMNET avoids to improve performance, the GLMNET implementation may not converge [62,70]. A case in point is that [71] provides two numerical experiments in which GLMNET does not converge.

Other coordinate descent algorithms have been developed to compute solutions to non-smooth MaxEnt models directly (see, e.g., [64,72]) but they are also generally non-parallelizable and often lack robustness and good convergence properties. Finally, another issue is that many coordinate descent algorithms, including GLMNET, were designed for sparse MaxEnt models (e.g., the elastic net and group lasso regularized MaxEnt models described in Section 2.3) and those algorithms depend on the sparsity of the model to converge quickly [54]. It would be desirable to have fast optimization methods for when the feature mapping (13) is dense, as this often occurs in practice.

First-order methods. First-order optimization algorithms such as the forward–backward splitting algorithm are popular because they are robust and can provably compute $ϵ$-approximate solutions of (3) with an optimal rate of convergence. However, as discussed in detail at the end of the previous subsection, achieving this rate of convergence requires fine-tuning the stepsize parameters using an accurate estimate of the largest singular value of the feature mapping (13), and this estimate is typically computationally expensive for large matrices. This problem makes classical first-order optimization algorithms inefficient and impractical in estimating probability densities from large-scale MaxEnt models.

Summary. The current state-of-the-art algorithms for estimating probability densities from MaxEnt models either produce unreliable numerical results, lack scalable parallelism, or scale poorly in size. These shortcomings in terms of robustness and efficiency make it challenging to use non-smooth, large-scale MaxEnt models in applications without access to adequate and costly computational resources. The next section presents novel, efficient, and robust accelerated NPDHG optimization methods that address these shortcomings.

4. Main Results

We describe here our approach for computing solutions to the generalized MaxEnt problem (3) using accelerated NPDHG optimization methods [38]. In addition to the standing assumptions on the potential function $H^{*}$, we assume we can compute efficiently the proximal operator of the convex conjugate of the potential function

$$\underset{\mathit{w}\in {\mathbb{R}}^m}{\mathrm{arg\; min}}\left\{\lambda H\left(\mathit{w}\right)+{\displaystyle \frac{1}{2}}{∥\mathit{w}-\widehat{\mathit{w}}∥}_2^2\right\},$$

for every $\lambda >0$ and $\widehat{\mathit{w}}\in {\mathbb{R}}^m$. These assumptions are satisfied for most potential functions used in practice, including for the potential functions described in Section 2.3.

4.1. Methodology

We start with the saddle-point formulation (4) of the generalized MaxEnt estimation problem (3). Based on the work two of the authors provided in [38], we propose to split the infimum and supremum in the saddle-point problem (4) using an iterative NPDHG scheme that alternates between a nonlinear proximal ascent step based on the Kullback–Leibler divergence and a linear proximal descent step.

More precisely, let ${\tau }_0>0$ and ${\sigma }_0>0$ be two stepsize parameters satisfying the inequality

$${\tau }_0{\sigma }_0\left(\underset{j\in \{1,\dots ,n\}}{max}{∥\mathbf{\Phi }\left(j\right)∥}_2^2\right)⩽1,$$

(14)

let ${\theta }_0=0$, let ${\mathit{w}}_{-1}={\mathit{w}}_0\in \mathrm{int}\mathrm{dom}H$, let ${\mathit{z}}_0\in {\mathbb{R}}^m$, and define the initial probability distribution $p_0\in {\Delta }_n$ through the constant ${\mathit{z}}_0$ via

$$p_0\left(j\right)={\displaystyle \frac{p_{\mathrm{prior}}\left(j\right)e^{\langle {\mathit{z}}_0,\mathbf{\Phi }\left(j\right)\rangle }}{{\sum }_{j=1}^n{p}_{\mathrm{prior}}\left(j\right)e^{\langle {\mathit{z}}_0,\mathbf{\Phi }\left(j\right)\rangle }}}$$

(15)

for each $j\in \{1,\dots ,n\}$. Our proposed NPDHG iterative scheme computes the iterates

$$\begin{array}{cc}\hfill p_{k+1}& =\underset{p\in {\Delta }_n}{\mathrm{arg\; min}}\left\{{\tau }_k{D}_{\mathrm{KL}}(p\parallel p_{\mathrm{prior}})-{\tau }_k〈{\mathit{w}}_k+{\theta }_k({\mathit{w}}_k-{\mathit{w}}_{k-1}),{\mathbb{E}}_p[\mathbf{\Phi }]〉+D_{\mathrm{KL}}(p\parallel p_k)\right\}\hfill \\ \hfill {\widehat{\mathit{w}}}_{k+1}& ={\mathit{w}}_k+{\sigma }_k({\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p_{k+1}}[\mathbf{\Phi }]),\hfill \\ \hfill {\mathit{w}}_{k+1}& =\underset{\mathit{w}\in {\mathbb{R}}^m}{\mathrm{arg\; min}}\left\{t{\sigma }_{k}H\left(\mathit{w}\right)+{\displaystyle \frac{1}{2}}{∥\mathit{w}-{\widehat{\mathit{w}}}_{k+1}∥}_2^2\right\},\hfill \\ \hfill {\theta }_{k+1}& =1/\sqrt{1+{\tau }_k},{\tau }_{k+1}={\theta }_{k+1}{\tau }_k\mathrm{and}{\sigma }_{k+1}={\sigma }_k/{\theta }_{k+1}.\hfill \end{array}$$

(16)

According to ([38], Proposition 4.1), the sequence of iterates $p_k$ converges strongly to the unique solution of the generalized MaxEnt problem (3). Moreover, for any $t⩾0$ and a given tolerance level $ϵ>0$, the scheme (16) provably computes an $ϵ$-approximate solution $p^{ϵ}\left(t\right)$ of the generalized MaxEnt model (3) in $O(1/\sqrt{ϵ})$ time. This rate of convergence is, without further smoothness assumptions on the potential function $H^{*}$, the best achievable rate of convergence with respect to the Nesterov class of optimal first-order methods [39].

The key element in this scheme is the choice of the Kullback–Leibler divergence as a nonlinear proximal step in the first line of (16). We use it for two reasons: First, because the Kullback–Leibler divergence already appears in the saddle-point problem (4). This allows us to compute $p_{k+1}$ explicitly. Indeed, thanks to the choice of initial probability distribution (15), we have

$$p_{k+1}\left(j\right)={\displaystyle \frac{p_{\mathrm{prior}}\left(j\right)e^{\langle {\mathit{z}}_{k+1},\mathbf{\Phi }\left(j\right)\rangle }}{{\sum }_{j=1}^n{p}_{\mathrm{prior}}\left(j\right)e^{\langle {\mathit{z}}_{k+1},\mathbf{\Phi }\left(j\right)\rangle }}}\mathrm{with}{\mathit{z}}_{k+1}={\displaystyle \frac{1}{1+{\tau }_k}}\left({\mathit{z}}_k+{\tau }_k({\mathit{w}}_k+{\theta }_k({\mathit{w}}_k-{\mathit{w}}_{k-1}))\right)$$

(17)

for each $j\in \{1,\dots ,n\}$. See Appendix A for the derivation of (17).

Second, because the Kullback–Leibler divergence is 1-strongly convex with respect to the ${\ell }_1$ norm, that is,

$$D_{\mathrm{KL}}(p\parallel p_k)⩾{\displaystyle \frac{1}{2}}{∥p-p_k∥}_1^2.$$

This fact follows from a fundamental result in information theory known as Pinsker’s inequality [47,48,49,50,51]. In particular, this means that scheme (16) alternates between solving a strongly convex problem over the space $({\mathbb{R}}^n,{∥·∥}_1)$ and a concave problem over the space $({\mathbb{R}}^m,{∥·∥}_2)$. The choice of these spaces is significant, for the induced operator norm ${∥·∥}_{\mathrm{op}}$ of the linear operator $\mathit{A}$ defined in (13) becomes

$${∥\mathit{A}∥}_{\mathrm{op}}=\underset{{∥p∥}_1=1}{sup}{∥\mathit{A}p∥}_2=\underset{j\in \{1,\dots ,n\}}{max}{∥\mathbf{\Phi }\left(j\right)∥}_2.$$

This operator norm offers two crucial advantages: First, it can be computed in optimal $\Theta \left(mn\right)$ time, or better if the features ${\{\mathbf{\Phi }\left(j\right)\}}_{j=1}^n$ have structure, e.g., if they are sparse. This means that the stepsize parameters ${\tau }_0$ and ${\sigma }_0$ of the NPDHG scheme can be computed in (14), with equality, in optimal $\Theta \left(mn\right)$ time. This is important because, in classical first-order optimization methods, we typically require a precise estimate of the largest singular value of the feature mapping (13), namely the number

$${∥\mathit{A}∥}_2=\underset{{∥\mathbf{x}∥}_2=1}{sup}{∥\mathit{A}p∥}_2,$$

to fine-tune the stepsize parameters to gain an optimal convergence rate. However, as discussed in detail in Section 3.1, this estimate is computationally expensive for large matrices due to its prohibitive cubic cost of $O\left(\min (m^{2}n,m{n}^2)\right)$ operations. In contrast, our NPDHG scheme (16) does not suffer from this computational bottleneck and therefore scales well. Second, the operator norm ${∥\mathit{A}∥}_{\mathrm{op}}$ can be significantly smaller than the estimate ${∥\mathit{A}∥}_2$, hence allowing for bigger stepsize parameters to further speed up convergence. (An easy calculation yields ${max}_{{∥\mathit{p}∥}_2=1}{∥\mathit{A}\mathit{p}∥}_2⩾{max}_{j\in \{1,\dots ,n\}}{∥\mathbf{\Phi }\left(j\right)∥}_2$.)

Summary. To solve the generalized MaxEnt estimation problem (3) and its dual problem (5), let ${\tau }_0>0$ and ${\sigma }_0>0$ be two stepsize parameters satisfying inequality (14), let ${\theta }_0=0$, let ${\mathit{w}}_{-1}={\mathit{w}}_0\in \mathrm{int}\mathrm{dom}H$, let ${\mathit{z}}_0\in {\mathbb{R}}^m$, and define an initial probability distribution $p_0$ through ${\mathit{z}}_0$ via (15). Then, compute the iterates

$$\begin{array}{cc}\hfill {\mathit{z}}_{k+1}& =\left({\mathit{z}}_k+{\tau }_k({\mathit{w}}_k+{\theta }_k({\mathit{w}}_k-{\mathit{w}}_{k-1}))\right)/(1+{\tau }_k),\hfill \\ \hfill p_{k+1}\left(j\right)& ={\displaystyle \frac{p_{\mathrm{prior}}\left(j\right)e^{\langle {\mathit{z}}_{k+1},\mathbf{\Phi }\left(j\right)\rangle }}{{\sum }_{j=1}^n{p}_{\mathrm{prior}}\left(j\right)e^{\langle {\mathit{z}}_{k+1},\mathbf{\Phi }\left(j\right)\rangle }}}\mathrm{for}j\in \{1,\dots ,n\},\hfill \\ \hfill {\widehat{\mathit{w}}}_{k+1}& ={\mathit{w}}_k+{\sigma }_k({\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p_{k+1}}[\mathbf{\Phi }]),\hfill \\ \hfill {\mathit{w}}_{k+1}& =\underset{\mathit{w}\in {\mathbb{R}}^m}{\mathrm{arg\; min}}\left\{t{\sigma }_{k}H\left(\mathit{w}\right)+{\displaystyle \frac{1}{2}}{∥\mathit{w}-{\widehat{\mathit{w}}}_{k+1}∥}_2^2\right\},\hfill \\ \hfill {\theta }_{k+1}& =1/\sqrt{1+{\tau }_k},{\tau }_{k+1}={\theta }_{k+1}{\tau }_k\mathrm{and}{\sigma }_{k+1}={\sigma }_k/{\theta }_{k+1},\hfill \end{array}$$

(18)

until convergence is achieved. As all parameters and updates can be computed in $O\left(mn\right)$ time, for any $t⩾0$ and a given tolerance level $ϵ>0$, the overall complexity for computing an $ϵ$-approximate solution $p^{ϵ}\left(t\right)$ is $O(mn/\sqrt{ϵ})$.

4.2. Algorithm for Smooth Potential Functions

The iterative scheme (18) does not require the potential function $H^{*}$ to be smooth. If, however, the potential function $H^{*}$ is ${\gamma }_{H^{*}}$-smooth (equivalently, H is ${\displaystyle \frac{1}{{\gamma }_{H^{*}}}}$-strongly convex) for some ${\gamma }_{H^{*}}>0$, then we can modify the NPDHG iterative scheme (18) to achieve a linear rate of convergence. More precisely, let $t>0$ and introduce the stepsize parameters

$$\theta =1-{\displaystyle \frac{t}{2{\gamma }_{H^{*}}{∥\mathit{A}∥}_{\mathrm{op}}^2}}\left(\sqrt{1+{\displaystyle \frac{4{\gamma }_{H^{*}}{∥\mathit{A}∥}_{\mathrm{op}}^2}{t}}}-1\right),\tau ={\displaystyle \frac{1-\theta }{\theta }}\mathrm{and}\sigma ={\displaystyle \frac{{\gamma }_{H^{*}}\tau }{t}}.$$

(19)

Let ${\mathit{w}}_{-1}={\mathit{w}}_0\in \mathrm{int}\mathrm{dom}H$, let ${\mathit{z}}_0\in {\mathbb{R}}^m$, and define $p_0$ through ${\mathit{z}}_0$ via (15). Then, the explicit NPDHG iterative scheme is

$$\begin{array}{cc}\hfill {\mathit{z}}_{k+1}& =\left({\mathit{z}}_k+\tau ({\mathit{w}}_k+\theta ({\mathit{w}}_k-{\mathit{w}}_{k-1}))\right)/(1+\tau ),\hfill \\ \hfill p_{k+1}\left(j\right)& ={\displaystyle \frac{p_{\mathrm{prior}}\left(j\right)e^{\langle {\mathit{z}}_{k+1},\mathbf{\Phi }\left(j\right)\rangle }}{{\sum }_{j=1}^n{p}_{\mathrm{prior}}\left(j\right)e^{\langle {\mathit{z}}_{k+1},\mathbf{\Phi }\left(j\right)\rangle }}}\mathrm{for}j\in \{1,\dots ,n\},\hfill \\ \hfill {\widehat{\mathit{w}}}_{k+1}& ={\mathit{w}}_k+\sigma ({\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p_{k+1}}[\mathbf{\Phi }]),\hfill \\ \hfill {\mathit{w}}_{k+1}& =\underset{\mathit{w}\in {\mathbb{R}}^m}{\mathrm{arg\; min}}\left\{t\sigma H\left(\mathit{w}\right)+{\displaystyle \frac{1}{2}}{∥\mathit{w}-{\widehat{\mathit{w}}}_{k+1}∥}_2^2\right\}.\hfill \end{array}$$

(20)

According to ([38], Proposition 4.3), the sequences of iterates $p_k$ and ${\mathit{w}}_k$ converge strongly to the unique solution of the generalized MaxEnt estimation problem (3) and its dual problem (5). Moreover, for any $t>0$ and a given tolerance level $ϵ>0$, this scheme provably computes an $ϵ$-approximate solution $p^{ϵ}\left(t\right)$ of the generalized MaxEnt estimation problem (3) in $O\left(\log \right(1/ϵ\left)\right)$. This rate of convergence is the best achievable rate of convergence with respect to the Nesterov class of optimal first-order methods [39]. As all parameters and updates can be computed in $O\left(mn\right)$ time, the overall complexity for computing an $ϵ$-approximate solution $p^{ϵ}\left(t\right)$ is $O\left(mn\log \right(1/ϵ\left)\right)$.

5. Application to Wildfire Science

To illustrate the efficiency of our novel algorithms on large-scale problems, we present here an application to wildfire science. The problem at hand is to combine fire occurrence data with ecological data in a fixed geographical region to estimate the probability of fire occurrences as a function of ecological features. MaxEnt models achieve this goal by translating fire occurrence and ecological data into probabilities of fire occurrences and ecological features. This approach closely mirrors how MaxEnt models are used for modeling species’ geographic distributions [19,20,22,24,25,26,27,28]. Another related goal is to identify what ecological features correlate most with fire occurrences. This can be achieved using a sparse MaxEnt model, e.g., an elastic net or group lasso regularized MaxEnt model, to identify ecological features correlating significantly with fire occurrences.

For this application, we use the Western US MTBS-Interagency (WUMI) wildfire data set [45], which we describe in Section 5.1 below. We formulate the problem of combining the fire occurrence and ecological data from the WUMI wildfire data set into a MaxEnt estimation problem in Section 5.2. Using this data, we then fit the elastic net, (non-overlapping) group lasso, and ${\ell }_{\infty }$ MaxEnt models for a large number of hyperparameters weighting the regularization. We detail this fitting procedure and the explicit NPDHG algorithms for these MaxEnt models in Section 5.3. Following this, we compare in Section 5.4 the running times required to fit the aforementioned MaxEnt models using our NPDHG algorithms with the forward–backward splitting algorithm [40,65] and the STRUCTMAXENT2 coordinate descent algorithm of [41]. Finally, in Section 5.5, we interpret the results obtained from fitting the aforementioned MaxEnt models to the WUMI wildfire data set.

5.1. WUMI Wildfire Data Set

The Western US MTBS-Interagency wildfire data set [45] consists of all fires (⩾1 km²) from multiple state and federal agencies, supplemented by satellite observations of large fires (⩾4 km²) from the Monitoring Trends in Burn Severity (MTBS) program, in the continental United States west of 103° W longitude. For this application, we extracted all wildfires from the WUMI data set that occurred between 1984 and 2020 inclusive (accessed 18 May 2023). The locations of all fires used are shown in Figure 1.

Next, following the procedure outlined in [44], we overlay the fire locations on a $12\mathrm{km}\times 12\mathrm{km}$ grid to construct a data frame of prevailing climate, vegetation, topographic, and human-related features for each fire. Altogether we include a total of 35 potential fire-related features from various sources; we provide a summary of all features used in our analysis along with their sources in Appendix B. For each grid cell, we also provide an index identifying its Environmental Protection Agency (EPA) Level III ecoregion. Defined on the basis of roughly homogeneous climate and vegetation patterns, ecoregions are commonly used in the wildfire science literature to identify climate–fire relationships at a coarse spatial scale [29,73]. The full WUMI wildfire data set with associated fire-related features for all months from 1984 to 2020 is publicly available as part of the Stochastic Machine Learning for wildfire activity in the western US (SMLFire1.0) package [44].

5.2. Data Preprocessing

We construct a weakly informative prior [74,75] for incorporating existing knowledge of wildfire conducive environmental conditions in our MaxEnt model. First, we prepare a training data set of 10,000 fire occurrences and absences chosen randomly across all months between 1984 and 2020 correlated with the values of 35 fire-related features described in the previous section. The features are averaged over each calendar month (i.e., January, February, …) between 1984 and 2020. We then apply min–max scaling to each feature, ensuring that all features lie in the same range. Second, we construct two Random Forest (RF) models, one each for fires in dry (May–September) and wet (October–April) seasons respectively. We withhold 20% of the training data to tune the model hyperparameters, such that the optimal hyperparameters ensure a trade-off between model precision and recall. Next, we predict the fire probability for all grid cells with fire-related features using either the trained wet or dry season RF model depending on the calendar month. Since the MaxEnt algorithm assumes a presence-only framework, that is, the absence of fires in a grid cell does not imply a non-zero fire probability, we impute grid cells without a fire with probability $p_{\mathrm{nfire}}=\min \left(p_{\mathrm{fire}}\right)/10$. Last, we normalize the prior probability distribution to ensure that the fire probability across all grid cells sums up to one. For convenience, we represent the prior distribution as the normalized mean of 12 monthly fire probabilities predicted by the RF models in Figure 1.

To construct the empirical distribution $\widehat{\mathcal{D}}$, we divide the study region into its 18 different EPA level III ecoregions and weigh the relative frequencies of fire among the ecoregions using the following strategy. For each ecoregion $r\in \{1,\dots ,18\}$, let $n_{\mathrm{r},\mathrm{total}}$ denote the total number of grid cells in ecoregion r and let $n_{\mathrm{r},\mathrm{fire}}$ denote the total number grid cell in ecoregion r where at least one fire was recorded. In addition, let $Z={\sum }_{r=1}^{18}{\displaystyle \frac{n_{\mathrm{r},\mathrm{fire}}}{n_{\mathrm{r},\mathrm{total}}}}$ denote the sum of the proportions of grid cells where at least one fire was recorded among all ecoregions. Then, we compute the empirical probability at cell $j\in \{1,\dots ,n\}$, $\widehat{\mathcal{D}}\left(j\right)$, as follows:

$$\widehat{\mathcal{D}}\left(j\right)={\displaystyle \frac{1}{Z}}\left\{\begin{array}{cc}0\hfill & \mathrm{if}\mathrm{no}\mathrm{fire}\mathrm{was}\mathrm{recorded}\mathrm{in}\mathrm{grid}\mathrm{cell}j,\hfill \\ 1/n_{\mathrm{r},\mathrm{total}}\hfill & \mathrm{if}\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{fire}\mathrm{was}\mathrm{recorded}\mathrm{in}\mathrm{grid}j\mathrm{and}\mathrm{the}\mathrm{grid}\mathrm{cell}j\mathrm{belongs}\mathrm{to}\mathrm{ecoregion}r\hfill \end{array}\right.$$

(21)

This construction gives the empirical distribution more weight to ecoregions where fires are more frequent and widespread.

5.3. Fitting Procedure and Algorithmic Setup

For the analysis, we fit six different MaxEnt models to the wildfire data: four elastic net MaxEnt models with parameters $\alpha =\{0.95,0.4,0.15,0.05\}$, a non-overlapping group lasso MaxEnt model, and an ${\ell }_{\infty }$-regularized MaxEnt model. For each model, we fit a regularization path of 141 hyperparameters as follows:

$$t^{\left(l\right)}=\left\{\begin{array}{cc}(1-l/100)t^{\left(0\right)}\hfill & \mathrm{from}l=0\mathrm{to}50\hfill \\ (0.5-(l-50)/200)t^{\left(0\right)}\hfill & \mathrm{from}l=51\mathrm{to}140,\hfill \end{array}\right.$$

where $t^{\left(0\right)}$ depends on the MaxEnt model and corresponds to the smallest hyperparameter for which the primal and dual solutions are equal to the prior distribution and zero. More precisely, for each model we set $(p^{\left(0\right)},{\mathit{w}}^{\left(0\right)})=(p_{\mathrm{prior}},\mathbf{0})$ and compute the sequence of solutions ${\left\{(p^{\left(l\right)},{\mathit{w}}^{\left(l\right)})\right\}}_{l=1}^{140}$ of the corresponding MaxEnt primal and dual problems (7) and (8) with hyperparameter $t=t^{\left(l\right)}$.

We chose these MaxEnt models to study the impact of the regularization on the primal and dual solutions as a function of the hyperparameters. In particular, for the elastic net and non-overlapping group lasso MaxEnt models, we are interested in identifying the set of features that are selected or discarded as a function of the sequence of hyperparameters ${\left\{t^{\left(l\right)}\right\}}_{l=0}^{140}$. In the parts below, we describe the value of $t^{\left(0\right)}$ and the NPDHG algorithm used for each model.

5.3.1. Elastic Net MaxEnt Models

For these MaxEnt models, the smallest hyperparameter for which the solutions to the primal and dual problems (7) and (8) are equal to the prior distribution and zero is

$$t^{\left(0\right)}={∥{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p_{\mathrm{prior}}}[\mathbf{\Phi }]∥}_{\infty }/\alpha .$$

This follows from the optimality condition

$${\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p^s}[\mathbf{\Phi }]-t(1-\alpha ){\mathit{w}}^s\in t\alpha \partial {∥·∥}_1{\mathit{w}}^s$$

(22)

and computing the smallest parameter t for which (22) is satisfied at $(p^s,{\mathit{w}}^s)=(p_{\mathrm{prior}},\mathbf{0})$.

For the elastic net MaxEnt model with $\alpha =0.95$, we use the NPDHG algorithm (18) (with sublinear convergence rate), while, for the models with $\alpha =\{0.4,0.15,0.05\}$, we use the NPDHG algorithm (20) (with linear convergence rate). Starting from $l=1$, we compute the pair of solutions $(p^{\left(l\right)},{\mathit{w}}^{\left(l\right)})$ to the primal and dual problems (7) and (8) at hyperparameter $t=t^{\left(l\right)}$ using the previously computed pair of solutions $(p^{(l-1)},{\mathit{w}}^{(l-1)})$ by setting the initial vectors ${\mathit{z}}_0={\mathit{w}}_0={\mathit{w}}^{(l-1)}$ in both NPDHG algorithms. For the stepsize parameters, we set ${\theta }_0=0$, ${\tau }_0=2$, and ${\sigma }_0=1/2{∥∥\mathit{A}∥∥}_{\mathrm{op}}^2$ in (18), and we set $\theta$, $\tau$, and $\sigma$ according to the formulas in (19) with ${\gamma }_{H^{*}}=1-\alpha$ in (20). We compute the update ${\mathit{w}}_{k+1}$ in both NPDHG algorithms using the classical soft thresholding operator [66,76,77]. Specifically, for any $\lambda >0$, $\widehat{\mathit{w}}\in {\mathbb{R}}^m$, and $i\in \{1,\dots ,m\}$ we have

$${\left[\underset{\mathit{w}\in {\mathbb{R}}^m}{\mathrm{arg\; min}}\left\{\lambda {∥\mathit{w}∥}_1+{\displaystyle \frac{1}{2}}{∥\mathit{w}-\widehat{\mathit{w}}∥}_2^2\right\}\right]}_i\equiv {\left[{\mathrm{shrink}}_1(\widehat{\mathit{w}},\lambda )\right]}_i=\{\begin{array}{cc}{\left[\widehat{\mathit{w}}\right]}_i-\lambda \hfill & \mathrm{if}{\left[\widehat{\mathit{w}}\right]}_i>\lambda ,\hfill \\ 0\hfill & \mathrm{if}|{\left[\widehat{\mathit{w}}\right]}_i|⩽\lambda ,\hfill \\ [\widehat{\mathit{w}}{]}_i+\lambda \hfill & \mathrm{if}{\left[\widehat{\mathit{w}}\right]}_i<-\lambda ,\hfill \end{array}$$

and so, for every $\lambda >0$, $\alpha \in [0,1]$, $\widehat{\mathit{w}}\in {\mathbb{R}}^m$ and $i\in \{1,\dots ,m\}$ we have

$${\left[\underset{\mathit{w}\in {\mathbb{R}}^m}{\mathrm{arg\; min}}\left\{\lambda \left(\alpha {∥\mathit{w}∥}_1+{\displaystyle \frac{(1-\alpha )}{2}}{∥\mathit{w}∥}_2^2\right)+{\displaystyle \frac{1}{2}}{∥\mathit{w}-\widehat{\mathit{w}}∥}_2^2\right\}\right]}_i={\displaystyle \frac{{\left[{\mathrm{shrink}}_1(\widehat{\mathit{w}},\lambda \alpha )\right]}_i}{1+\lambda (1-\alpha )}}.$$

Finally, we let the NPDHG algorithms run for at least 40 iterations before checking for convergence. We stop the NPDHG algorithms when the optimality condition (22) is satisfied within some tolerance ${10}^{-5}$:

$${∥{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p_k}[\mathbf{\Phi }]-t(1-\alpha ){\mathit{w}}_k∥}_{\infty }⩽t\alpha (1+{10}^{-5}).$$

5.3.2. Non-Overlapping Group Lasso MaxEnt Model

For this MaxEnt model, we divide the features into five disjoint groups of features, as described in Appendix B. Then, the smallest hyperparameter for which the solutions to the primal and dual problems (7) and (8) are equal to the prior distribution and zero is

$$t^{\left(0\right)}=\underset{g\in \{1,\dots ,5\}}{max}\left\{{∥{\mathbb{E}}_{\widehat{\mathcal{D}}}\left[{\mathbf{\Phi }}_g\right]-{\mathbb{E}}_p\left[{\mathbf{\Phi }}_g\right]∥}_{2,g}/\sqrt{m_g}\right\},$$

where $m_g$ is the number of features in the gth group. This follows from the optimality condition

$${\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p^s}[\mathbf{\Phi }]\in \bigcup _{g=1}^5\left\{t\sqrt{m_g}\partial {∥·∥}_{2,g}\right\}{\mathit{w}}_g^s$$

(23)

and computing the smallest parameter t for which (23) is satisfied at $(p^s,{\mathit{w}}^s)=(p_{\mathrm{prior}},\mathbf{0})$.

For this model, we use the NPDHG algorithm (18). Starting from $l=1$, we compute the pair of solutions $(p^{\left(l\right)},{\mathit{w}}^{\left(l\right)})$ to the primal and dual problems (10) and (9) at hyperparameter $t=t^{\left(l\right)}$ using the previously computed pair of solutions $(p^{(l-1)},{\mathit{w}}^{(l-1)})$ by setting the initial vectors ${\mathit{z}}_0={\mathit{w}}_0={\mathit{w}}^{(l-1)}$. For the stepsize parameters, we set ${\theta }_0=0$, ${\tau }_0=2$, and ${\sigma }_0=1/2{∥∥\mathit{A}∥∥}_{\mathrm{op}}^2$. We compute the update ${\mathit{w}}_{k+1}$ in (18) using the following proximal operator formula: for every group $g\in \{1.\dots ,5\}$, $\lambda >0$ and $\widehat{\mathit{w}}\in {\mathbb{R}}^m$,

$$\underset{\mathit{w}\in {\mathbb{R}}^m}{\mathrm{arg\; min}}\left\{\lambda \sqrt{m_g}{∥\mathit{w}∥}_{2,g}+{\displaystyle \frac{1}{2}}{∥\mathit{w}-\widehat{\mathit{w}}∥}_{2,g}^2\right\}=max\left(0,1-\lambda \sqrt{m_g}/{∥\widehat{\mathit{w}}∥}_2\right)\widehat{\mathit{w}}.$$

Finally, we let the NPDHG algorithms run for at least 40 iterations before checking for convergence. We stop the NPDHG algorithms when the optimality condition (23) is satisfied within some tolerance ${10}^{-5}$:

$$\underset{g\in \{1,\dots ,5\}}{max}{∥{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p_k}[\mathbf{\Phi }]∥}_{2,g}⩽t(1+{10}^{-5}).$$

5.3.3. ${\ell }_{\infty }$-Regularized MaxEnt Model

For this MaxEnt model, the smallest hyperparameter for which the solutions to the primal and dual problems (7) and (8) are equal to the prior distribution and zero is

$$t^{\left(0\right)}={∥{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p_{\mathrm{prior}}}[\mathbf{\Phi }]∥}_1.$$

This follows from the optimality condition

$${\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p^s}[\mathbf{\Phi }]\in t\partial {∥·∥}_{\infty }\left({\mathit{w}}^s\right)$$

(24)

and computing the smallest parameter t for which (24) is satisfied at $(p^s,{\mathit{w}}^s)=(p_{\mathrm{prior}},\mathbf{0})$.

For this model, we use the NPDHG algorithm (18). Starting from $l=1$, we compute the pair of solutions $(p^{\left(l\right)},{\mathit{w}}^{\left(l\right)})$ to the primal and dual problems (11) and (12) at hyperparameter $t=t^{\left(l\right)}$ using the previously computed pair of solutions $(p^{(l-1)},{\mathit{w}}^{(l-1)})$ by setting the initial vectors ${\mathit{z}}_0={\mathit{w}}_0={\mathit{w}}^{(l-1)}$. For the stepsize parameters, we set ${\theta }_0=0$, ${\tau }_0=2$, and ${\sigma }_0=1/2{∥∥\mathit{A}∥∥}_{\mathrm{op}}^2$. Finally, we compute the update ${\mathit{w}}_{k+1}$ in (18) using Moreau’s decomposition ([78], Theorem 3.2.5): for every $\lambda >0$ and $\widehat{\mathit{w}}\in {\mathbb{R}}^m$,

$$\underset{\mathit{w}\in {\mathbb{R}}^m}{\mathrm{arg\; min}}\left\{\lambda {∥\mathit{w}∥}_{\infty }+{\displaystyle \frac{1}{2}}{∥\mathit{w}-\widehat{\mathit{w}}∥}_2^2\right\}=\widehat{\mathit{w}}-\underset{{∥\mathit{w}∥}_1⩽\lambda }{\mathrm{arg\; min}}{\displaystyle \frac{1}{2}}{∥\mathit{w}-\widehat{\mathit{w}}∥}_2^2.$$

The second term on the right amounts to projecting $\widehat{\mathit{w}}$ on the ${\ell }_1$ ball of radius $\lambda$. There are fast algorithms for doing do; we use Algorithm 1 described in [79]. Finally, we let the NPDHG algorithms run for at least 40 iterations before checking for convergence. We stop the NPDHG algorithms when the optimality condition (24) is satisfied within some tolerance ${10}^{-5}$:

$${∥{\mathbb{E}}_{\widehat{\mathcal{D}}}[\mathbf{\Phi }]-{\mathbb{E}}_{p_k}[\mathbf{\Phi }]∥}_1⩽t(1+{10}^{-5}).$$

5.4. Comparison of Timings

In this section, we compare the run times of our NPDHG algorithms with two state-of-the-art optimization algorithms for solving non-smooth MaxEnt models: the forward–backward splitting algorithm (specifically, Algorithm 5 in [65]; see also [40,66]) and the STRUCTMAXENT2 coordinate descent algorithm from [41,64]. All numerical experiments were performed on a single core Intel(R) Core(TM) i7-10750H CPU @ 2.60 GHz.

We initially chose the GLMNET implementation [26,33] in MATLAB 2023b over STRUCTMAXENT2 for the numerical comparisons, but we found that GLMNET produced unreliable numerical solutions when compared to both the NPDHG and forward–backward splitting algorithms. We also tried using GLMNET’s implementation in the R language, but to no avail. We think this problem arises because, as discussed in Section 3.2, the GLMNET algorithm approximates a MaxEnt model as a logistic regression model and then invokes a coordinate descent method tailored to logistic regression to approximate the solution to the MaxEnt model. Our observations suggest this approach does not work well for our data set. In contrast, we found that the STRUCTMAXENT2 algorithm produced correct numerical results.

For the forward–backward splitting algorithm, the stepsize parameters were set to $1/{∥\mathit{A}∥}_2$ and 1 (corresponding to $\tau$ and $t_0$ in Algorithm 5 of [65]) and, for computing the pair of solutions $(p^{\left(l\right)},{\mathit{w}}^{\left(l\right)})$ at hyperparameter $t^{\left(l\right)}$, the initial iterate was set to ${\mathit{w}}^{(l-1)}$. In addition, for the elastic net MaxEnt models, the acceleration quantity $q=(1-\alpha )t^{\left(l\right)}/({∥\mathit{A}∥}_2+(1-\alpha )t^{\left(l\right)})$ was employed. We used the stopping criteria of the NPDHG algorithms for the forward–backward splitting algorithms. For the coordinate descent algorithm, we modified the STRUCTMAXENT2 algorithm from ([41], pp. 305–306) to make it applicable to the elastic net penalty. For computing the pair of solutions $(p^{\left(l\right)},{\mathit{w}}^{\left(l\right)})$ at hyperparameter $t^{\left(l\right)}$, the initial iterate was set to ${\mathit{w}}^{(l-1)}$. We did not use the STRUCTMAXENT2 algorithm for the non-overlapping group lasso or ${\ell }_{\infty }$-regularized MaxEnt models, as it was not designed for these MaxEnt models. We used the stopping criteria of the NPDHG algorithms for the STRUCTMAXENT2 algorithm.

Table 1. Timings results (in seconds) for fitting the MaxEnt models described in Section 5.3. All timings are averaged over five runs.

	STRUCTMAXENT2	Forward–Backward Splitting	NPDHG
Elastic net ($\alpha =0.95$)	5562.19	4208.01	365.55
Elastic net ($\alpha =0.40$)	1018.73	1407.22	113.53
Non-overlapping group lasso	N/A	3036.38	278.14
${\ell }_{\infty }$-regularization	N/A	2534.65	289.98

shows the average timings for computing the entire regularization path of the WUMI wildfire data set using the coordinate descent, forward–backward splitting, and NPDHG algorithms. All timings were averaged over five runs and, for the forward–backward splitting and NPDHG algorithms, they include the time required to compute all the stepsize parameters. All algorithms were implemented in MATLAB.

The NPDHG algorithm outperformed both the forward–backward splitting and coordinate descent algorithms by at least one order of magnitude. In particular, we observed that the NPDHG algorithm required far fewer iterations to achieve convergence compared to both the forward–backward splitting and STRUCTMAXENT2 algorithms. This difference is because the stepsize parameters for the NPDHG algorithm were much larger compared to either the forward–backward splitting or the STRUCTMAXENT2 algorithm. Indeed, the stepsize parameters for the NPDHG and forward–backward splitting algorithms are inversely proportional to the norms ${∥A∥}_{\mathrm{op}}$ and ${∥A∥}_2$, and for the wildfire data set these were ${∥\mathit{A}∥}_{\mathrm{op}}\approx 3.30$ and ${∥A∥}_2\approx 854.08$. Thus, larger stepsize parameters were permitted thanks to the Kullback–Leibler divergence term in the NPDHG algorithm, enabling a major speedup gain.

5.5. Analysis of the MaxEnt Regularization Paths and Estimated Fire Probabilities

As a final validation step for the NPDHG algorithm, we use the fitted MaxEnt models to compute the normalized mean fire probability in each grid cell for all calendar months between 1984 and 2020. The probabilities are visualized in Figure 2 and Figure 3. In each case, we have chosen the prior distribution as a benchmark for our spatial plots of fire probability.

The spatial fire probabilities for $\alpha =\{0.95,0.40,0.15,0.05\}$ are shown in Figure 2. The range of $\alpha$ values roughly corresponds to varying the regularization from a purely $l_1$ norm ($\alpha =1$) to a purely $l_2$ norm ($\alpha =0$). For each value of $\alpha$, we also consider the evolution of the spatial fire probability as we vary the hyperparameter t along the regularization path, or equivalently include additional features while fitting the MaxEnt model to wildfire data. Broadly, we observe that, for a fixed value of $t/t_{\max }$, the ratio of predicted to prior fire probability decreases in sharpness as $\alpha$ decreases. On the other hand, for a fixed $\alpha$, decreasing $t/t_{\max }$ values enables the model to make sharper distinctions between grid cells with high and low fire probability as evidenced by the sharper contrast between the prior and predicted fire probabilities. We also note a similar pattern in Figure 3 for the MaxEnt models with non-overlapping group lasso (corresponding roughly to the elastic net case with $\alpha =1$) and the $l_{\infty }$-MaxEnt models, all of which converge to the empirical distribution quicker than any of the elastic net cases.

In Figure 4, we show the cumulative number of non-zero coefficients at fixed intervals along the regularization path for different $\alpha$ values. The plot helps in visualizing the $t/t_{\max }$ values at which new features are introduced in the elastic net MaxEnt model, with the dashed vertical lines indicating the first time a feature from a new group is selected. Across all $\alpha$ values, we find that features appear in the same order, with fire weather features being selected first, followed by topography, vegetation, human, and antecedent features. We tabulate the final set of non-zero features at the end of the regularization path for $\alpha =\{0.95,0.05\}$ across various groups in

Table 2. List of non-zero features at the end of the regularization path for two elastic net penalty parameters, organized by different feature groups. See Appendix B for additional description of the selected features.

Feature Group	$\mathit{\alpha }=0.95$	$\mathit{\alpha }=0.05$
Fire weather	Tmax	Tmax	VPD
	Prec	Prec	Tmin
	Wind	Wind	${\mathrm{VPD}}_{\max 3}$
	FM1000	FM1000	${\mathrm{VPD}}_{\max 7}$
	${\mathrm{Tmin}}_{\max 3}$	${\mathrm{Tmin}}_{\max 3}$	${\mathrm{Tmax}}_{\max 3}$
	Lightning	Lightning	${\mathrm{Tmax}}_{\max 7}$
		${\mathrm{Tmin}}_{\max 7}$	${\mathrm{SWE}}_{\max }$
Topography	Slope	Slope	Southness
	Southness
Vegetation	Grassland	Biomass	Forest
		Shrub
Human	Camp_dist	Camp_dist	Road_dist
	Camp_num	Camp_num
	Urban	Urban
	Popdensity	Popdensity
Antecedent	${\mathrm{AvgPrec}}_{2\mathrm{mo}}$	${\mathrm{AvgPrec}}_{2\mathrm{mo}}$	${\mathrm{AntPrec}}_{\mathrm{lag}2}$
	${\mathrm{AvgVPD}}_{3\mathrm{mo}}$	${\mathrm{AntPrec}}_{\mathrm{lag}1}$

. These selected features are in good agreement with physical models of wildfire occurrence [42,43] as well as previous statistical analyses of wildfire drivers [29,30,31,44].

6. Discussion

In this paper, we have introduced novel first-order NPDHG algorithms that overcome the shortcomings of state-of-the-art algorithms for training large-scale, non-smooth MaxEnt models. The crucial ingredient common to our algorithms is the Kullback–Leibler divergence. Using it over the classical proximal operator makes it possible to train large-scale and non-smooth MaxEnt models much more efficiently than the state of the art. In particular, all stepsize parameters and iterations in our algorithms can be calculated on the order $O\left(mn\right)$ operations, improving on the complexity bound of $O\left(\min (m^{2}n,m{n}^2)\right)$ operations for computing the optimal stepsize parameters of classical first-order optimization methods, such as the linear PDHG or forward–backward splitting methods. As a consequence, for a given tolerance level $ϵ>0$, our algorithms provably compute solutions using on the order of $O(mn/\sqrt{ϵ})$ or $O(mn/\log (1/ϵ\left)\right)$ operations, the order depending on the smoothness of the MaxEnt model and which is optimal with respect to the Nesterov class of optimal first-order methods [39]. Moreover, the computational bottleneck consists of matrix–vector multiplications, which can be trivially parallelized, and so our algorithms exhibit scalable parallelism.

Finally, we have shown that the strong convexity of the Kullback–Leibler divergence with respect to the ${\ell }_1$ norm allows for significantly larger stepsize parameters, thereby speeding up the convergence rate of our algorithms. This was, in particular, observed in Section 5, when we applied our algorithms to fit the WUMI wildfire data set [45] on several non-smooth MaxEnt models to estimate the probabilities of fire occurrences as a function of ecological features. Our algorithms outperformed the state-of-the-art forward–backward splitting and coordinate descent STRUCTMAXENT2 algorithms by at least one order of magnitude. They also yielded results that are in good agreement with physical models of wildfire occurrence [42,43] as well as previous statistical analyses of wildfire drivers [29,30,31,44]. Future work will explore the scalability of our algorithms for modeling daily scale wildfire probability [80]. It will also explore how the choices of different informative priors affect the MaxEnt models using prediction accuracy as a metric.

We expect our algorithms to provide efficient methods for solving non-smooth MaxEnt models that arise in large-scale machine learning applications beyond the wildfire application explored in this paper. An interesting future direction, albeit tangential to the focus of this work, would be to prove learning guarantees for the MaxEnt problem (3) for arbitrary choices of potential functions, similarly to [19,64], for the choice of ${\ell }_1$ and ${\ell }_2$ norms as potential functions. This would provide some means to assess the “quality” of the probability distribution estimated from the MaxEnt problem. Another direction that would be interesting and impactful is to extend our algorithms to continuous regularized MaxEnt models and prove rigorously that they work: interesting because the continuous version of the MaxEnt problem is an infinite dimensional problem, which makes this problem more technically challenging, impactful because such an algorithm would enable a much broader class of probability distributions to be used in MaxEnt modeling.