Exploring Flatter Loss Landscape Surface via Sharpness-Aware Minimization with Linear Mode Connectivity

Abstract

The Sharpness-Aware Minimization (SAM) optimizer connects flatness and generalization, suggesting that loss basins with lower sharpness are correlated with better generalization. However, SAM requires manually tuning the open ball radius, which complicates its practical application. To address this, we propose a method inspired by linear connectivity, using two models initialized differently as endpoints to automatically determine the optimal open ball radius. Specifically, we introduce distance regularization between the two models during training, which encourages them to approach each other, thus dynamically adjusting the open ball radius. We design an optimization algorithm called ’Twin Stars Entwined’ (TSE), where the stopping condition is defined by the models’ linear connectivity, i.e., when they converge to a region of sufficiently low distance. As the models iteratively reduce their distance, they converge to a flatter region of the loss landscape. Our approach complements SAM by dynamically identifying flatter regions and exploring the geometric properties of multiple connected loss basins. Instead of searching for a single large-radius basin, we identify a group of connected basins as potential optimization targets. Experiments conducted across multiple models and in varied noise environments showed that our method achieved a performance on par with state-of-the-art techniques.

1. Introduction

In recent years, deep neural networks have advanced machine learning, with over-parameterization increasing fitting capabilities and expanding complex solution spaces, risking overfitting [1]. This reduces saddle points and increases local minima [2], each with varying generalization abilities [3]. Current studies have shown a positive correlation between loss basin flatness and generalization [4,5,6,7,8,9]. Flatness can be described using a Hessian matrix, which represents the second-order derivative [10]. However, computing a Hessian matrix during gradient descent in a large network incurs significant computational costs. To improve the efficiency of gradient descent, machine learning typically relies on first-order optimizers, such as stochastic gradient descent (SGD), which, however, lack awareness of the flatness direction [11].

Reconstructing the flatness perception of SGD is a key challenge for improving optimization. The Sharpness-Aware Minimization (SAM) strategy [9] perturbs the gradient to approximate second-order curvature information [12]. Specifically, SAM minimizes the worst-case loss within a perturbation region of radius $\rho$ around the current solution. However, due to the diverse shapes of the loss landscapes across different networks and datasets [13], selecting an appropriate $\rho$ is challenging. A larger $\rho$ may result in the inability to perceive loss basins, as there may not exist a loss basin in the loss landscape that meets the required radius, leading to divergence and higher training losses.

To address this, Algorithm 1 ASAM [14] normalizes the solution space before setting $\rho$, aiming to stabilizing optimization. GSAM [15] further normalizes gradients to mitigate uncertainty, allowing $\rho$ to be set within a controlled range (typically between 0 and 1). However, manual tuning remains necessary. Inspired by the concept of linear mode connectivity [16], we propose an approach that models $\rho$ dynamically using two points, A and B, at the end point in linear mode connectivity of the solution space. This eliminates the need for manual tuning, making SAM-based optimization more adaptive and robust.

Algorithm 1 ASAM [14]

Input: DataSet S, Loss function L, Batch size b, Step size $\eta >0$, Neighborhood size $\rho >0$, Initial model ${\mathit{\theta }}_0^{init}$, Output: Model trained with ASAM Start at $\mathit{\theta }={\mathit{\theta }}_0^{init}$ repeat    Sample batch $\mathcal{R}=\left\{\left({\mathit{x}}_1,{\mathit{y}}_1\right),\dots \left({\mathit{x}}_b,{\mathit{y}}_b\right)\right\}$    Compute gradient ${\nabla }_{\mathit{\theta }}{L}_{\mathcal{B}}\left(\mathit{\theta }\right)$ of the batch’s training loss;    Compute $\widehat{\mathit{ϵ}}\left(\mathit{\theta }\right)=\rho T_{\mathit{\theta }}^2\nabla L_B\left(\mathit{\theta }\right)/{∥T_{\mathit{\theta }}\nabla L_B\left(\mathit{\theta }\right)∥}_2$    Compute gradient approximation for the SAM objective $\mathit{g}={\left.{\nabla }_w{L}_{\mathcal{B}}\left(\mathit{\theta }\right)\right|}_{\mathit{\theta }+\widehat{\mathit{ϵ}}\left(\mathit{\theta }\right)}$    Update weights: ${\mathit{\theta }}_{t+1}={\mathit{\theta }}_t-\eta \mathit{g}$    $t=t+1$ until converged return ${\mathit{\theta }}_t$

In summary, our heuristic strategy gradually reduces the distance between two points in the solution space, guiding the search toward flatter regions that improve the generalization. By incorporating connectivity awareness into sharpness-aware minimization, we enable the optimization process to target these flatter regions, making the search more effective. Linear mode connectivity improves our understanding of the relationships between minima, allowing the algorithm to converge to connected minima along linear modes, thereby enhancing the optimization. Experimental results confirmed that our method achieved improved convergence and better generalization.

We propose a heuristic training strategy aimed at generating linearly connected mode endpoints [16]. The radius $\rho$ of the SAM method is modeled using two points, A and B, in the solution space, to avoid manually setting the hyperparameter $\rho$. Specifically, we redefine the SAM open ball using A and B. The midpoint between A and B serves as the center, and the radius $\rho$ is defined as half the distance between them (see Figure 1). In each training iteration, our heuristic algorithm optimizes the models corresponding to points A and B, while adding a distance regularization constraint that simulates the SAM optimization process. To ensure connectivity, we employ a subspace learning strategy [17]. Specifically, our algorithm initializes the optimization process at the center of points A and B in each iteration and applies subspace optimization techniques to maintain a linearly connected path [17,18]. The goal of our strategy is to gradually reduce the distance between A and B, guiding them to converge toward the same flat local minimum basin or neighboring local minimum basins with low loss barriers, as illustrated in Figure 1. This process is akin to a binary star system, where A and B behave like stars in space, mutually influencing each other’s gravitational pull, thus reducing the distance between them over time.

Our contributions are summarized as follows:

• We explored how to achieve linear mode connectivity in sharpness-aware optimization, and we propose that connectivity can be leveraged to dynamically adjust the radius $\rho$ in SAM optimization, removing the need for manual tuning.
• We developed a heuristic search algorithm that combines flatness optimization with linear connectivity. This algorithm simulates the gradual convergence of two points in the solution space, ensuring that when the search stops, the two solutions exhibit both linear connectivity and flatness within a geometric range that includes their neighbors.
• We conducted comprehensive validation experiments, including testing across multiple network architectures and visualization studies, confirming the effectiveness of our approach.

2. Related Works

2.1. Sharpness-Aware Minimization

Recent works have suggested that flat local minima are associated with better generalization ability [19,20,21,22]. However, flatness does not directly equate to generalization, and flat regions may even exist in areas of high loss. Furthermore, some studies have argued that flatness does not exhibit a strong statistical correlation with generalization performance [23]. The Probably Approximately Correct (PAC) theory defines generalization, and SAM [9] uses the upper bound of generalization as the optimization objective. Subsequent works have focused on gradient estimation methods using this upper bound, including scale-invariant approaches [14], and zero-order and first-order flatness estimation methods [15], among others. SAM has also been extended beyond Euclidean space to statistical manifolds [24] and Riemannian manifolds [25], where the optimization process was adapted to these more complex spaces. Potential improvements to SAM optimization could involve refining the gradient vector estimation, leading to more accurate directions for gradient descent optimization.

2.2. (Linear) Mode Connectivity

Seeking linear mode connectivity during training implies partial flatness, meaning the existence of low-loss regions along the direction of a specific vector [26,27,28,29]. In sharpness-aware optimization, a surface represents the local minimum, with a defined center and radius, ensuring linear connectivity between any two points within the basin. This implies that the line connecting the two furthest points in the basin corresponds to the basin’s diameter in the flatness model of the local minimum, as discussed by [17,29]. Methods for identifying connectivity include improvements in gradient descent (SGD) [26,30] and heuristic approaches [16,31]. Building on previous work, the process begins by selecting two points that are separated by significant loss barriers [16]. Then, the distance between them is gradually reduced while maintaining flatness, until a termination point is reached, signifying linear connectivity. This process helps identify a broad flattened basin. When combined with SAM optimization using a fixed hyperparameter $\rho$, this resembles an implicit pinching strategy, aimed at locating a region with a broader $\rho$.

3. Preliminary

3.1. Sharpness-Aware Minimization

The classification model operates within the joint probability distribution space $(\mathcal{X},\mathcal{Y})$, where $\mathcal{X}$ represents the sample space and $\mathcal{Y}$ denotes the label space. The training set, denoted as $\mathcal{D}$, adheres to the distribution $(\mathcal{X},\mathcal{Y})$, and consists of n samples, $\mathcal{S}={\left\{\left(x_i,y_i\right)\right\}}_{i=1}^n$, independently sampled from $\mathcal{D}$. The parameters of the model, denoted by $\mathit{\theta }\in \mathrm{\Theta }\subseteq {\mathbb{R}}^d$, are used to model the region around $\mathit{\theta }$ with an open ball $B(\mathit{\theta },\rho )$, where $\rho >0$ is the radius and $\mathit{\theta }$ is the center for the open ball B in the Euclidean space, i.e., $B(\mathit{\theta },\rho )=\left\{{\mathit{\theta }}^{\prime }:{∥\mathit{\theta }-{\mathit{\theta }}^{\prime }∥}_p<\rho \right\}$.

The training loss, $L_{\mathcal{S}}\left(\mathit{\theta }\right)={\sum }_{i=1}^{n}l\left({\mathbf{y}}_i,f\left({\mathbf{x}}_i;\mathit{\theta }\right)\right)/n$, is the average of individual losses over the training set. The generalization gap, which compares the expected loss $L_{\mathcal{D}}\left(\mathit{\theta }\right)={\mathbb{E}}_{(\mathbf{x},\mathbf{y})\sim D}\left[l(\mathbf{y},f(\mathbf{x};\mathit{\theta }))\right]$ to the training loss $L_{\mathcal{S}}\left(\mathit{\theta }\right)$, measures the ability of the model to generalize to unseen data.

Sharpness-Aware Minimization (SAM) aims to minimize the following PAC-Bayesian generalization error upper bound:

$$\begin{array}{c}\hfill L_{\mathcal{D}}\left(\mathit{\theta }\right)\le \underset{{\parallel \mathit{ϵ}\parallel }_p\le \rho }{max}{L}_{\mathcal{S}}(\mathit{\theta }+\mathit{ϵ})+h\left({\displaystyle \frac{{\parallel \mathit{\theta }\parallel }_2^2}{{\rho }^2}}\right).\end{array}$$

(1)

With optimization operators for $L_{\mathcal{S}}$, there is an empirical risk of minimization. h is a strictly increasing function that can be replaced by a norm function. Sharpness-Aware Minimization (SAM) [9] employs a one-step backward approach to obtain gradient approximation:

$$\begin{array}{c}\hfill \widehat{\mathit{ϵ}}\left(\mathit{\theta }\right)=\rho {\nabla }_{\mathit{\theta }}{L}_{\mathcal{S}}\left(\mathit{\theta }\right)/∥{\nabla }_{\mathit{\theta }}{L}_{\mathcal{S}}\left(\mathit{\theta }\right)∥.\end{array}$$

(2)

Then, SAM computes the gradient with respect to the perturbed model $\mathit{\theta }+\widehat{\mathit{ϵ}}$ for the second step update:

$$\begin{array}{c}\hfill {\left.{\nabla }_{\mathit{\theta }}{L}_{\mathcal{S}}^{SAM}\left(\mathit{\theta }\right)\approx {\nabla }_{\mathit{\theta }}{L}_{\mathcal{S}}\left(\mathit{\theta }\right)\right|}_{\mathit{\theta }+\widehat{\mathit{ϵ}}}.\end{array}$$

(3)

The detailed SAM is presented in Algorithm 2.

Algorithm 2 SAM [9]

Input: DataSet S, Loss function L, Batch size b, Step size $\eta >0$, Neighborhood size $\rho >0$, Initial model ${\mathit{\theta }}_0^{init}$ Output: Model trained with SAM Start at $\mathit{\theta }={\mathit{\theta }}_0^{init}$ repeat    Sample batch $\mathcal{R}=\left\{\left({\mathbf{x}}_1,{\mathbf{y}}_1\right),\dots \left({\mathbf{x}}_b,{\mathbf{y}}_b\right)\right\}$    Compute gradient ${\nabla }_{\mathit{\theta }}{L}_{\mathcal{B}}\left(\mathit{\theta }\right)$ of the batch’s training loss;    Compute $\widehat{\mathit{ϵ}}\left(\mathit{\theta }\right)=\rho {\nabla }_{\mathit{\theta }}{L}_{\mathcal{S}}\left(\mathit{\theta }\right)/∥{\nabla }_{\mathit{\theta }}{L}_{\mathcal{S}}\left(\mathit{\theta }\right)∥$    Compute gradient approximation for the SAM objective $\mathit{g}={\left.{\nabla }_w{L}_{\mathcal{B}}\left(\mathit{\theta }\right)\right|}_{\mathit{\theta }+\widehat{\mathit{ϵ}}\left(\mathit{\theta }\right)}$    Update weights: ${\mathit{\theta }}_{t+1}={\mathit{\theta }}_t-\eta \mathit{g}$    $t=t+1$ until converged return ${\mathit{\theta }}_t$

3.2. Analysis of Sharpness-Aware Minimization

An optimal perturbation radius $\rho$ is crucial for Sharpness-Aware Minimization (SAM) convergence. The ideal ${\rho }_{\mathrm{target}}$ should balance the local curvature and flat region geometry, but an explicit search is challenging due to non-hyperspherical flat regions and their network-dependent variability [13]. In non-convex optimization, geometric uncertainty forces a reliance on trial-and-error for $\rho$ selection.

To address the challenge of hyperparameter selection, we propose a principled method for selecting $\rho$ from the perspective of linear mode connectivity, to enable a more efficient screening. Through observation, we identified that flat regions in the loss landscape inherently satisfy linear connectivity. Based on this, we present the following theory.

Theorem 1.

The optimal ${\rho }_{\mathit{target}}$ is the maximal diameter of linearly mode connected regions [16], defined as:

$${\rho }_{\mathit{target}}=sup\left\{\rho >0|\forall {\mathit{\theta }}_A,{\mathit{\theta }}_B\in B(\mathit{\theta },\rho ),\mathit{linear}\mathit{mode}\mathit{connectivity}\mathit{holds}\right\}.$$

(4)

Theorem 1 ensures optimization within flat regions, but non-convexity complicates direct implementation. We instead use backward reasoning, approximating ${\rho }_{\mathrm{target}}$ via nonlinear mode connectivity.

Theorem 2.

For any pair of parameters ${\mathit{\theta }}_A,{\mathit{\theta }}_B$. If ${\mathit{\theta }}_A$ and ${\mathit{\theta }}_B$ exhibit a loss barrier (i.e., are not linear mode connected), they cannot coexist in a ball $B(\mathit{\theta },\rho )$ of any radius $\rho >0$, also including ${\rho }_{\mathit{target}}$.

Although Theorem 2 cannot directly lead us to an appropriate solution, it can help us eliminate distracting solutions.

Building on both insights, we propose an implicit $\rho$-search algorithm that operationalizes linear connectivity as a dynamic selection heuristic. First, we reformulate the optimization of SAM by modeling the endpoints of its hyperspherical perturbation as linear mode connectivity (Section 4). Subsequently, we introduce a Twin Stars Entwined (TSE) algorithm to co-optimize these paired models, steering the gradient search toward flatter regions (Section 5).

4. Modeling the SAM with Endpoints

4.1. Definition

We redefine the SAM open ball by taking the midpoint C of two points A and B, which are the endpoints of a line passing through the center of the sphere, as the center, and the half-distance between these two points as the radius $\rho$. Mathematically, the center C and radius $\rho$ are expressed as

$$\begin{array}{c}\hfill C={\displaystyle \frac{A+B}{2}},\rho ={\displaystyle \frac{\parallel A-B\parallel }{2}},\end{array}$$

(5)

where center C is a virtual center as the ball center definition in SAM. The optimization goal of SAM is to minimize the loss function on the set of points $\mathbf{z}\in B(C,\rho )$, where $B(C,\rho )$ is the open ball centered at C with radius $\rho$:

$$\begin{array}{c}\hfill L_{\mathcal{S}}^{SAM}\approx \underset{\forall \mathbf{z}\in B(C,\rho )}{\mathrm{avgmin}}\underset{\rho }{max}L\left(\mathbf{z}\right).\end{array}$$

(6)

However, Equation (6) cannot be directly optimized. The reason for this is that individually optimizing A and B cannot guarantee they reside within the same SAM (Sharpness-Aware Minimization) conceptual ball. In other words, when optimized separately, A and B may converge to different basins, due to factors like the initialization and optimization algorithms.

Inspired by Theorem 2, we change the target of the optimization process to achieve linear mode connectivity. In our framework, this is equivalent to minimizing the loss between points A and B directly. The premise is to ensure linear mode connectivity [16], as in (7).

4.2. Train Subspace

We introduce the target radius ${\rho }_{\mathrm{target}}$, defined as the distance at which the path between points A and B is linearly connected. This means that there exists a straight line segment connecting A and B, and all points along this segment lie within a relatively flat region of the loss function. Mathematically, we define linear connectivity as follows:

$$\begin{array}{c}\hfill \mathrm{Linear}\mathrm{Mode}\mathrm{Connectivity}:\exists \gamma \left(\alpha \right)\in {\mathbb{R}}^d,\alpha \in [0,1]\end{array}$$

(7)

$$\begin{array}{c}\hfill \mathrm{such}\mathrm{that}\gamma \left(0\right)=A,\gamma \left(1\right)=B,\forall \alpha \in [0,1],\parallel \nabla {\mathcal{L}}_{\mathcal{S}}{\left(\gamma \left(\alpha \right)\right)\parallel }_2=0,\end{array}$$

(8)

$$\begin{array}{c}\hfill \mathrm{where}\gamma \left(\alpha \right)=(1-\alpha )A+\alpha B,\alpha \in [0,1].\end{array}$$

(9)

In the context of linear mode connectivity, the ideal linear interpolation between two optima ${\theta }_1$ and ${\theta }_2$ should satisfy $\parallel \nabla {\mathcal{L}}_{\mathcal{S}}{\left(\gamma \left(\alpha \right)\right)\parallel }_2=0$ for all $\alpha$, implying a connected low-loss path in the parameter space. Subspace learning [17] provides a framework to guarantee linear mode connectivity, defined as

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{subspace}}={\mathbb{E}}_{\alpha \sim \mathcal{U}[0,1]}\left[{∥\nabla {\mathcal{L}}_{\mathcal{S}}\left(\gamma \left(\alpha \right)\right)∥}_2\right].\end{array}$$

(10)

However, subspace learning requires a large number of weighted sampling operations on the two weights. In practical applications, the model definition needs to be significantly modified, which is not convenient for integration. Note that SAM provides a linear connectivity guarantee around the hypersphere $\rho$ of radius from Theorem 1. Inspired by this, we combine subspace learning with the SAM process to explore a flatter basin.

5. Proposed Algorithm

To better find flatter local minimum basins, SAM (Sharpness-Aware Minimization) and subspace learning provide several factors. The details are illustrated in Figure 2.

To provide an intuitive description of our motivation, we present the following facts:

1.

Ideal SAM guarantees connectivity for the convergent solution and a hyperball centered at this solution with radius $\rho$.

2.

A larger $\rho$ to attempt convergence to flatter regions. However, if $\rho$ is too large, the algorithm may oscillate due to the absence of basins in the solution space that meet the current $\rho$ requirement.

3.

Based on Theorem 2, the connectivity of two points A and B in the solution space determines whether they lie within the hyperball of ideal SAM.

4.

If SAM optimization with radius $\rho$ is applied to endpoints A and B separately, under ideal conditions, A and B will converge, and if the distance between A and B is less than or equal to $2\rho$, then A and B are linearly connected.

5.

When Fact 4 is satisfied, the flat regions guaranteed by A and B are adjacent. The combined flat region guaranteed by A and B is larger than either of them individually, and thus can be regarded as a flatter region.

6.

If the line connecting A and B is treated as the diameter of a virtual ball, and Fact 4 is satisfied, then the virtual ball formed by A and B may also conform to a larger ideal SAM hyperball with radius $\rho$.

Summarizing the above facts, we propose a new optimization approach to implicitly obtain a larger flat region. We define an optimization process where we model the line connecting A and B as a virtual ball with radius ${\rho }^{\prime }$. Assuming A and B are optimized using $\rho$-SAM and exhibit nonlinear connectivity upon convergence, we use distance regularization to bring A and B closer, and restart training until linear connectivity is achieved, at which point we obtain a larger flat region. We further model the process of A and B approaching each other as a process with T iterative steps, as detailed in Section 6. During optimization, the target radius is determined by the iterates $A_t$ and $B_t$ at step t, defined as

$$\begin{array}{cc}\hfill {\rho }_{\mathrm{target}}& =min\left\{\rho \mid \rho \ge \parallel A_t-B_t{\parallel }_2\right\},\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.\parallel \nabla {\mathcal{L}}_{\mathcal{S}}{\left(\gamma \left(\alpha \right)\right)\parallel }_2=0,\alpha \sim \mathcal{U}[0,1],\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & \gamma \left(0\right)=A_t,\gamma \left(1\right)=B_t.\hfill \end{array}$$

(13)

As optimization progresses, $A_t$ and $B_t$ move closer, until they enter the same or adjacent loss basins, satisfying

$$\begin{array}{c}\hfill \underset{t\to T}{lim}\parallel A_t-B_t{\parallel }_2={\rho }_{\mathrm{target}},\parallel \nabla {\mathcal{L}}_{\mathcal{S}}\left(A_t\right){\parallel }_2={\parallel \nabla {\mathcal{L}}_{\mathcal{S}}\left(B_t\right)\parallel }_2=0.\end{array}$$

(14)

Our algorithm has the potential to converge if $\rho$-SAM converges. If the manual setting of $\rho$ for the SAM algorithm converges, there exists at least one region in the solution space where both points A and B can converge simultaneously. The extreme case is when A and B collapse into a single point as they approach each other.

Our algorithm offers two key advantages. First, our algorithm eliminates the need for laborious manual tuning of $\rho$—instead of exhaustively searching for an optimal value, it only requires an initial $\rho$ that ensures SAM convergence, then automatically expands the search for flatter regions. Second, our algorithm inherently avoids poor solutions by design: supported by convergence analysis and Facts 5–6, the method stabilizes SAM while providing a provable lower-bound guarantee on solution quality.

6. Twin Stars Entwined (TSE) Problem

6.1. Problem Definition

The problem mentioned in Section 5 which we aim to solve involves identifying a pair of local minima, $({\mathit{\theta }}_t^A,{\mathit{\theta }}_t^B)$, in the solution space at step t, and then locating the next pair of local minima, $({\mathit{\theta }}_{t+1}^A,{\mathit{\theta }}_{t+1}^B)$, at step $t+1$. We require that the distance between these two minima decreases with each step, meaning that $Dist({\mathit{\theta }}_{t+1}^A,{\mathit{\theta }}_{t+1}^B)\le Dist({\mathit{\theta }}_t^A,{\mathit{\theta }}_t^B)$, where $Dist(·)$ represents the Euclidean distance. If we view the discovered pairs of local minima as a trajectory over time, the sequence $\{({\mathit{\theta }}_0^A,{\mathit{\theta }}_0^B),\dots ,({\mathit{\theta }}_t^A,{\mathit{\theta }}_t^B)\}$ can be imagined as two stars, A and B, gradually drawing closer to each other in the vast universe of the solution space. We refer to this problem as the Twin Stars Entwined (TSE) problem.

We propose a toy optimization approach for the Twin Stars Entwined (TSE) problem, defined by the following optimization problems:

$$\begin{array}{c}\hfill \underset{{\mathit{\theta }}_{t+1}^A}{min}{\alpha }^A\parallel {\mathit{\theta }}_{t+1}^A-{\mathit{\theta }}_t^A{\parallel }_2^2+(1-{\alpha }^A){\parallel {\mathit{\theta }}_{t+1}^A-{\mathit{\theta }}_t^B\parallel }_2^2\end{array}$$

(15)

$$\begin{array}{c}\hfill \underset{{\mathit{\theta }}_{t+1}^B}{min}{\alpha }^B\parallel {\mathit{\theta }}_{t+1}^B-{\mathit{\theta }}_t^A{\parallel }_2^2+(1-{\alpha }^B){\parallel {\mathit{\theta }}_{t+1}^B-{\mathit{\theta }}_t^B\parallel }_2^2.\end{array}$$

(16)

In this toy optimization approach, the current solutions ${\mathit{\theta }}_t^A$ and ${\mathit{\theta }}_t^B$ serve as anchor points to determine the next positions. The updates to the solutions, ${\mathit{\theta }}_{t+1}^A$ and ${\mathit{\theta }}_{t+1}^B$, are guided by their corresponding values in the previous step, ${\mathit{\theta }}_t^A$ and ${\mathit{\theta }}_t^B$. For example, setting ${\alpha }^A=0.9$ will make the optimal solution ${\mathit{\theta }}_{t+1}^A$ close to ${\mathit{\theta }}_t^A$. Similarly, if ${\alpha }^B=0.1$, the resulting optimal solution ${\mathit{\theta }}_{t+1}^B$ will be closer to ${\mathit{\theta }}_t^B$. As the iterations progress, the toy optimization process forms a trajectory where the two solutions gradually converge towards each other.

We employ weight importance evaluation to enhance the toy TSE optimization. Taking endpoint A as an example, we aim to minimize the distance between ${\mathit{\theta }}_{t+1}^A$ and ${\mathit{\theta }}_t^A$. It is crucial to apply weighted importance when evaluating distances. Based on the dropout feasibility assumptions and the lottery hypothesis [28,32], changes in redundant parameters do not impact model performance. We adopt the weight importance evaluation method used in continual learning tasks, specifically utilizing the Fisher Information Matrix [18,33]. To simplify, we only consider the diagonal elements of the Fisher matrix, assuming parameter independence. The Fisher matrix is computed as follows:

$$\begin{array}{c}\hfill F\left({\mathit{\theta }}_{\mathit{i}}\right)={\displaystyle \frac{1}{\left|S\right|}}\sum _{j=1}^{\left|S\right|}{\left({\displaystyle \frac{\partial logp\left({\mathbf{y}}_j\mid {\mathbf{x}}_j;\mathit{\theta }\right)}{\partial {\mathit{\theta }}_{\mathit{i}}}}\right)}^2.\end{array}$$

(17)

We rewrite the toy optimization from Equations (11) and (12) as a loss function:

$$\begin{array}{cc}\hfill L^{Dis}(\mathit{\theta },{\mathit{\theta }}_A^{\prime },{\mathit{\theta }}_B^{\prime },\alpha )& =\alpha \parallel F\left({\mathit{\theta }}_A^{\prime }\right)(\mathit{\theta }-{\mathit{\theta }}_A^{\prime }){\parallel }_2^2\hfill \\ \hfill & +(1-\alpha )(\parallel F\left({\mathit{\theta }}_B^{\prime }\right)(\mathit{\theta }-{\mathit{\theta }}_B^{\prime }){\parallel }_2^2.\hfill \end{array}$$

(18)

Here, $\mathit{\theta }$ represents the current gradient descent point in the solution space, while ${\mathit{\theta }}_A^{\prime }$ and ${\mathit{\theta }}_B^{\prime }$ are the outputs from the previous round. Weight importance introduces flexibility into the search space. If the weight importance is zero, this allows the parameter to change freely, enabling backpropagation to refine the model without disrupting existing knowledge. This flexibility is vital because we cannot ensure the existence of a stopping condition between the points in the current step.

Weight importance enhances the flexibility of TSE but does not guarantee generalization. By incorporating TSE into SAM optimization, we improve sensing and ensure better generalization. However, SAM struggles with perceiving loss barriers [16], as its forward steps may bypass loss barriers and move directly to another basin of loss-consistent contours. The perception of relationships between basins involves two key aspects. If a low-loss barrier path connects two basins, the gradient descent should follow this path [16,17,34]. If no such path exists, the optimizer must escape the current local minimum. Assuming a connected path between two global minima [2], a low-energy point [16] can guide the gradient descent. Thus, we can reformulate the optimization problem as

$$\begin{array}{c}\hfill \underset{\mathit{\theta }}{min}\underset{{\parallel \mathit{ϵ}\parallel }_2\le \rho }{max}{L}_{\mathcal{S}}(\mathit{\theta }+\mathit{ϵ})+\beta L^{Dis}(\mathit{\theta }+\mathit{ϵ},{\mathit{\theta }}_A^{\prime },{\mathit{\theta }}_B^{\prime },\alpha ).\end{array}$$

(19)

This process is repeated independently twice at each step. In the first step, gradient descent searches for ${\mathit{\theta }}_{t+1}^A$, aiming to stay close to ${\mathit{\theta }}_t^A$ with $\alpha =0.9$. In the second step, gradient descent searches for ${\mathit{\theta }}_{t+1}^B$, aiming to stay close to ${\mathit{\theta }}_t^B$ with $\alpha =0.1$. The details of each step are outlined in Algorithm 3.

Algorithm 3 TSE-SAM

Input: DataSet S, Loss function L, Batch size b, Step size $\eta >0$, Neighborhood size $\rho >0$, Initial model ${\mathit{\theta }}_0^{init}$, Pre-step models ${\mathit{\theta }}^A$ and ${\mathit{\theta }}^B$, trade off factor $\alpha$ Output: Model trained with TSE Start at $\mathit{\theta }={\mathit{\theta }}_0^{init}$, parameter set $\left[\mathit{\theta }\right]=[\mathit{\theta },{\mathit{\theta }}^A,{\mathit{\theta }}^B]$ repeat    Sample batch $\mathcal{R}=\left\{\left(x_1,y_1\right),\dots \left(x_b,y_b\right)\right\}$    Compute gradient ${\nabla }_{\mathit{\theta }}{L}_{\mathcal{B}}\left(\mathit{\theta }\right)$ and ${\nabla }_{\mathit{\theta }}{L}_{\mathcal{B}}^{Dis}(\left[\mathit{\theta }\right],\alpha )$ of the batch’s training loss;    Joint Gradient $\nabla {\widehat{L}}_{\mathcal{B}}\left(\mathit{\theta }\right)=\nabla L_{\mathcal{B}}\left(\mathit{\theta }\right)+\nabla L_{\mathcal{B}}^{Dis}(\left[\mathit{\theta }\right],\alpha )$    Compute $\widehat{\mathit{ϵ}}\left(\mathit{\theta }\right)=\rho {\displaystyle \frac{T_{`}^2\nabla {\widehat{L}}_{\mathcal{B}}\left(\mathit{\theta }\right)}{{∥T_{`}\nabla {\widehat{L}}_{\mathcal{B}}\left(\mathit{\theta }\right)∥}_2}}$    Compute gradient approximation for the SAM objective $\mathit{g}={\left.{\nabla }_w{L}_{\mathcal{B}}\left(\mathit{\theta }\right)\right|}_{\mathit{\theta }+\widehat{\mathit{ϵ}}\left(\mathit{\theta }\right)}$    Update weights: ${\mathit{\theta }}_{t+1}={\mathit{\theta }}_t-\eta \mathit{g}$    $t=t+1$ until converged return ${\mathit{\theta }}_t$

6.2. Iterative Strategy for Twin Stars Entwined (TSE) Problem

We design a iterative search process to solve the Twin Stars Entwined (TSE) problem. Initially, we randomly initialize ${\mathit{\theta }}_0^{init}$. We apply two training strategies, so that the gradient trajectories starting from ${\mathit{\theta }}_0^{init}$ can move in two different directions and converge to two distinct solutions. We do not start iterations from two separate random initializations to accelerate convergence. The initialization is defined as follows:

$$\begin{array}{c}\hfill {\mathit{\theta }}_0^A=SAM({\mathit{\theta }}_0^{init},S)\end{array}$$

(20)

$$\begin{array}{c}\hfill {\mathit{\theta }}_0^B=ASAM({\mathit{\theta }}_0^{init},S).\end{array}$$

(21)

After obtaining the two converged models A and B, we select the center of the ball as the starting point for the next round of the search. This idea is inspired by subspace optimization [17], aiming to also achieve low loss at the interpolation points. It is defined as follows:

$$\begin{array}{c}\hfill {\mathit{\theta }}_{t+1}^{init}={\displaystyle \frac{{\mathit{\theta }}_t^A+{\mathit{\theta }}_t^B}{2}}.\end{array}$$

(22)

If the stopping condition is not met, we consider the radius $\rho$ represented by the current models A and B to still be too large. Using A and B from the previous iteration as anchor points, we conduct a new round of search, defined as follows:

$$\begin{array}{c}\hfill {\mathit{\theta }}_{t+1}^A=TSE({\mathit{\theta }}_t^{init},{\mathit{\theta }}_t^A,{\mathit{\theta }}_t^B,{\alpha }^A)\end{array}$$

(23)

$$\begin{array}{c}\hfill {\mathit{\theta }}_{t+1}^B=TSE({\mathit{\theta }}_t^{init},{\mathit{\theta }}_t^A,{\mathit{\theta }}_t^B,{\alpha }^B).\end{array}$$

(24)

The object of our heuristic is to ensure that $dist({\mathit{\theta }}_{t+1}^A,{\mathit{\theta }}_{t+1}^B)<dist({\mathit{\theta }}_t^A,{\mathit{\theta }}_t^B)$, $dist(·)$ is the distance between two models. TSE has demonstrated empirical effectiveness in achieving this objective.

Stop condition Under the objective set for TSE to stop in flatter fields, as discussed in Section 4, the onset of linear mode connectivity serves as the stopping criterion for an iterative search. We measure linear mode connectivity using linear interpolation, defined as

$$\begin{array}{c}\hfill {\mathit{\theta }}_i={\mathit{\theta }}^A+{\displaystyle \frac{i}{N}}({\mathit{\theta }}^B-{\mathit{\theta }}^A).\end{array}$$

(25)

We derive a set $\left\{{\mathit{\theta }}_i\right|i=0,1,\dots ,N\}$ comprising N elements and compute the average and variance of each model’s loss on the train set. Subsequently, we employ the variance as a stop criterion. Essentially, our iterative search halts when the variance falls below a threshold $\delta$. Otherwise, our iterative search will conclude upon reaching the maximum iteration step I, as outlined in Algorithm 4:

$$\begin{array}{c}\mu ({\mathit{\theta }}^A,{\mathit{\theta }}^B)={\displaystyle \frac{1}{N}}\sum _{i=0}^N{L}_{\mathcal{S}}\left({\mathit{\theta }}_i\right)\hfill \end{array}$$

(26)

$$\begin{array}{c}\sigma ({\mathit{\theta }}^A,{\mathit{\theta }}^B)=\sqrt{\sum _{i=0}^N{(L_{\mathcal{S}}\left({\mathit{\theta }}_i\right)-\mu ({\mathit{\theta }}^A,{\mathit{\theta }}^B))}^2}.\hfill \end{array}$$

(27)

Algorithm 4 Iterative Search

Input: DataSet S, Max Rounds I, Trade Off Factor ${\alpha }^A$ and ${\alpha }^B$, Threshold $\delta$ Output:${\mathit{\theta }}^A$, ${\mathit{\theta }}^B$ Initialize weights ${\mathit{\theta }}_0^{init}$ Get initial point A ${\mathit{\theta }}_0^A=SAM({\mathit{\theta }}_0^{init},S)$ Get initial point B ${\mathit{\theta }}_0^B=ASAM({\mathit{\theta }}_0^{init},S)$ for $i=0$ to $I-1$ do    Get next step initial point ${\mathit{\theta }}_{t+1}^{init}=({\mathit{\theta }}_t^A+{\mathit{\theta }}_t^B)/2$    Get point A:    ${\mathit{\theta }}_{t+1}^A=TSE({\mathit{\theta }}_{t+1}^{init},S,{\mathit{\theta }}_t^A,{\mathit{\theta }}_t^B,{\alpha }^A)$    Get point B:    ${\mathit{\theta }}_{t+1}^B=TSE({\mathit{\theta }}_{t+1}^{init},S,{\mathit{\theta }}_t^A,{\mathit{\theta }}_t^B,{\alpha }^B)$    if $\sigma ({\mathit{\theta }}_{t+1}^A,{\mathit{\theta }}_{t+1}^B)<\delta$ then      Stop loop    end if end for return ${\mathit{\theta }}_t^A,{\mathit{\theta }}_t^B$

7. Experiment Setup

7.1. Image Classification: CIFAR-10/100

To evaluate the effectiveness of our proposed approach, we compared TSE with three baseline optimizers: Stochastic Gradient Descent (SGD), Sharpness-Aware Minimization (SAM) [9], Adaptive SAM (ASAM) [14], and GAM [15]. These methods served as baselines to assess the improvements introduced by our approach in terms of optimization stability and generalization performance. To ensure reproducibility, we retained the original hyperparameter settings without modification. We extended the baseline code from sam-main (Baseline code of CIFAR-10/100 was sourced from the GitHub repository sam-main: https://github.com/davda54/sam) (accessed on 1 February 2025) by adding the TSE training pipeline for datasets CIFAR-10/100 and the architecture definitions of resnet [35], wideresnet [36] and pyramidnet [37]. Our code is available at https://github.com/Timiain/TSE.git (accessed on 1 February 2025).

In the training setup, the same as the sam-main default setting, SAM with $\rho$ was set to 0.05 and Adaptive SAM(ASAM) with $\rho$ was set to 2, as the default in original paper. The experiment was conducted on the CIFAR-10 and CIFAR-100 datasets. The base optimizer was the Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.9 and using a linearly decreasing learning rate scheduler. The initial learning rate was 0.1, and L2 regularization with a weight decay of 0.0005 was used to enhance generalization. Label smoothing was applied with a smoothing factor of 0.1. The batch size was set to 128, and the model was trained for 200 epochs for each iteration, with a dropout rate of 0.0. Note that SAM-family is an optimization framework whose first step is to add a forward perturbed gradient and in a second step generate new a gradient after the first step update. The specific strategy for updating the model’s learnable parameters was implemented by the base optimizer. The experiment was run in a single Titan X gpu environment.

Then, we divided the training process into two stages, which were the Initial and TSE stages. In the initial stage, we specially trained models A and B with the SAM and ASAM optimizers. In the TSE stage, we set $I=5$ as the maximum number of iterations. In each iteration, new models of A and B were trained with hyperparameters and initial steps consistent with the github sam-main setting. Our key modifications were as follows:

• The first-step perturbation was replaced with the perturbation defined by the TSE Algorithm 3.
• The training loss was updated to Equation (19), which incorporates an additional EWC regularization term.

7.2. Semantic Segmentation: Vaihingen

To validate the adaptability of our TSE algorithm, we selected a remote sensing semantic segmentation task, which was distinct from image classification. For ease of replication, we selected the Vaihingen dataset [38] along with its state-of-the-art (SOTA) baselines: UnetFormer [39] and LSKNet [40]. Firstly, UnetFormer and LSKNet are UNet-like architectures. They blend self-attention mechanisms with convolutional networks. UnetFormer comprises a CNN encoder and a decoder that integrates a Swin-like [41] Transformer block and CNN structures in each decoder layer. LSKNet introduces a Large Kernel Selection (LSK) mechanism for dynamically selecting kernels in the encoder and UnetFormer decoder. Secondly, there was’ a significant visual disparity in the semantic content between the dataset for remote sensing semantic segmentation and the classification of the CIFAR dataset.

We used UNetFormer (Baseline code of UNetformer was sourced from the GitHub repository Unetformer: https://github.com/WangLibo1995/GeoSeg) (accessed on 1 February 2025) and LSKNet (Baseline code of LSKNet-UNetformer was sourced from the GitHub repository LSKNet: https://github.com/zcablii/GeoSeg) (accessed on 1 February 2025) for repository for replication. We inherited the hyperparameter and optimizer scheduler settings from the chosen baseline implementations. Specifically, the base optimizer was AdamW [42], and the learning rate scheduler was set to CosineAnnealingWarmRestarts. The total training epochs for each baseline was 105. We modified the baselines by adding SAM [9], ASAM [14], and TSE. Similarly, we divided the training process into the Initial and TSE stages, the same as in Section 8.1. We retained the recommended settings from the original paper, with $\rho =2$ for SAM and $\rho =0.05$ for ASAM, without manual tuning of hyperparameters $\rho$.

8. Experimental Results

8.1. Image Classification: CIFAR-10/100

In order to validate the performance of our method, we conducted verification on multiple network structures, including resnet [35], wideresnet [36], and pyramidnet [37]. We present the best results from our iterative iterations in

Table 1. Maximum test accuracies for SGD, SAM, ASAM, GAM, and TSE on CIFAR-10 dataset. The bold numbers indicate the best accuracy, signifying the best classification performance. The symbol ’*’ refers to the result from GAM [15]’s resnet101.

cifar10 (Acc%)	SGD	SAM	ASAM	GAM	TSE
resnet18	95.98	96.22	96.64	96.93	96.98
resnet50	96.2	95.98	96.55	97.35 *	97.12
wrn16-8	96.87	96.86	97.31	-	97.78
wrn28-2	95.78	95.68	96.4	95.94	96.55
wrn28-10	96.86	96.22	97.2	97.40	97.74
pyramid110-48	95.85	95.98	93.96	-	96.92
pyramid110-270	97.20	97.25	97.37	97.60	97.36

and

Table 2. Maximum test accuracies for SGD, SAM, ASAM, GAM, and TSE on CIFAR-100 dataset. The bold numbers indicate the best accuracy, signifying the best classification performance. The symbol ’*’ refers to the result from GAM [15]’s resnet101.

cifar100 (Acc%)	SGD	SAM	ASAM	GAM	TSE
resnet18	79.63	79.55	81.27	80.7	82.03
resnet50	78.74	79.56	81.38	83.2 *	83.45
wrn16-8	82.4	82.63	84.34	-	84.77
wrn28-2	77.7	77.84	79.64	77.89	80.73
wrn28-10	80.14	82.57	81.75	84.37	85.1
pyramid110-48	78.35	78.63	75.14	-	82.98
pyramid110-270	83.74	84.25	84.12	85.31	85.88

. As indicated in (11) and (12), we reproduced the results of SAM [9] and ASAM [14]. To evaluate the algorithm’s performance more accurately, we compared the latest results reported by GAM [15].

We divided the experimental setup into two stages. When training with the SAM and ASAM optimizers, we set the training epoch to 200, fixed the initial learning rate at 0.03, and used a linearly decreasing learning rate scheduler. The batch size was 128. For the $\rho$ hyperparameter, we did not search for better hyperparameters, and all experiments used a fixed $\rho$ value, with SAM using $\rho =0.05$ and ASAM using $\rho =2.0$, as suggested by the authors. Both the SAM and ASAM optimizers utilized Stochastic Gradient Descent (SGD) with a momentum of 0.9. We set $I=5$ as the maximum number of iterations. To ensure significant divergence from the initialized model during the iterative iteration steps, we initially set a high learning rate of 0.1 for each iteration.

Table 1 presents the maximum test accuracies of the different optimization algorithms (SGD, SAM, ASAM, GAM, and TSE) on the CIFAR-10 dataset. TSE generally performed the best across most models, especially on ResNet18, WRN16-8, WRN28-2, WRN28-10, and PyramidNet-110-48. GAM achieved the highest accuracy on ResNet50 and PyramidNet-110-270. Overall, TSE led in most cases, with GAM excelling for specific models like ResNet50 and PyramidNet-110-270. Table 2 shows the maximum test accuracies of the various optimization algorithms (SGD, SAM, ASAM, GAM, and TSE) on the CIFAR-100 dataset. Our TSE consistently outperformed the other algorithms for most models.

8.2. Semantic Segmentation: Vaihingen

We established a new baseline suitable for applying SAM-like optimization algorithms to the remote sensing semantic segmentation task. As shown in

Table 3. Maximum test mIoUs for AdamW, SAM, ASAM, GAM, and TSE on Vaihingen dataset. The bold numbers indicate the best mean Intersection over Union(mIoU), signifying the best semantic segmentation performance.

Vaihingen (mIoU%)	AdamW	SAM	ASAM	TSE
Unetformer	82.7	84.6	84.7	85.1
LSKNet-S	85.1	85.3	85.0	85.5

, TSE achieved a mIoU of 85.1% (85.07%) on UNetFormer, while TSE in LSKNet-S achieved a mIoU of 85.5% (85.47%). We observed that directly using SAM and ASAM with default parameters on a new task yielded varying performance across the different network architectures. Specifically, SAM showed positive effects both on UNetFormer and LSKNet, whereas ASAM had a negative impact on LSKNet-S. SAM and ASAM, without manual adjustment of hyperparameters, may not bring improvements in new scenarios, and our TSE process could mitigate this phenomenon.

8.3. Analysis of Iterative Search

In this section, we analyzed the behavior of our proposed iterative search algorithm.

8.3.1. Can $\rho$ Be Infinite?

To confirm our hypothesis that $\rho$ cannot be infinite, we conducted a hyperparameter search experiment on $\rho$. We set the grid search element $\rho =[{10}^{-3},{10}^{-2},{10}^{-1},{10}^0,{10}^1]$, as shown in Figure 3. When $\rho =10$, the results of both SAM and ASAM showed degradation, as in the report in [14]. The experiments in Figure 4 demonstrate that setting the radius $\rho$ to infinity was not feasible, indicating that the requirement to implicitly explore flatter surfaces is reasonable.

8.3.2. Results of Adaptive Radius

We proposed a heuristic strategy aimed at adaptively searching for an appropriate connectivity radius $\rho$ for each specific network architecture. We recorded the average Euclidean distance between corresponding layers of models A and B at each iteration, as illustrated in Figure 4. In Figure 4, the Y-axis represents the Euclidean distance between points A and B. It is observed that the inter-model distance gradually decreased as the iterations progressed. The Euclidean distance at the stopping criterion varied across the different models and dataset configurations. The experimental results also demonstrate the difficulty of manually setting the radius $\rho$ for SAM.

Another interesting finding is that, for each dataset, the ranking of the radius size at the stopping criterion across different networks was same as for the model architecture. For example, the largest radius was observed for ResNet18 and WRN16-8, while the smallest radius was found for PyramidNet-110-64. This finding provides a guide for further analysis of the geometric properties of the loss landscape associated with different network architectures.

8.3.3. Results of Linear Mode Connectivity

We employed visualization techniques, as described in [43], to illustrate the relationship between pairs of snapshots produced by our algorithm, as shown in Figure 5 and Figure 6. For this demonstration, we selected PyramidNet110 with a configuration of $\alpha =48$ as a representative example. Initially, both snapshots were trained from the same starting weight, allowing us to compare the differences between them.

When independently trained with SAM (Sharpness-Aware Minimization) and ASAM (Adaptive Sharpness-Aware Minimization), the resulting pairs of snapshots exhibited notable differences in their loss landscapes. Specifically, these snapshots showed distinct loss barriers, reflecting the impact of the different training methods. However, as the iteration steps progressed, the interpolation loss—the loss computed between the two snapshots generated by our algorithm—gradually decreased, indicating that the relationship between the snapshots became more aligned. This behavior suggests that our algorithm successfully minimized the loss discrepancy between the snapshots over time, leading to a smoother convergence in the training process.

We observed a distinct pattern in the behavior of the solution space. When two points were randomly selected within this space, the resulting loss was often quite high. This suggests that starting from different initialization points can lead to a reduction in loss during optimization. However, there is a significant risk that the solutions will converge to disconnected local minima, leading to suboptimal results. In this scenario, the interpolation between the two models will exhibit both high loss and low accuracy, as the solutions are not well-aligned.

We applied the proposed TSE search. As shown in Figure 4, when endpoints A and B gradually converged, we observed a consistent reduction in the interpolation loss between the two points. Initially, the loss was at the ${10}^4$ level, but by step 5 of the optimization process, it decreased to the ${10}^{-2}$ level, demonstrating significant progress. Throughout this process, the accuracy of the interpolated models remained consistently high, indicating a linear connectivity between the two models, which supported smoother transitions.

Furthermore, through detailed visualization analysis, the interpolation results provided additional confirmation that our model closely approximated a flatter loss surface. This suggests that the models, as they converged, approached a region of the solution space with less steep gradients, which was beneficial for stability and generalization.

8.3.4. Results of Model Diversity

In deep learning, the diversity of models can be measured by calculating the hierarchical cosine similarity of the model’s learnable parameters. Specifically, the learnable parameters of different models with the same architecture in various layers can be compared using cosine similarity to quantify their similarity in the solution space. As cosine similarity moves closer to 0, this indicates a low correlation at the parameter level [17,44], suggesting that the models may have learned complementary features, which helps improve the generalization ability of the ensemble model. In contrast, a higher cosine similarity may indicate significant redundancy among the models, which affects their diversity. By utilizing this metric, model architecture selection, ensemble learning methods, and distillation strategies can be optimized to enhance the overall performance.

We visualize the similarity of the endpoints of our search strategy with Figure 7, Figure 8 and Figure 9. We calculated the cosine similarity of the parameters at each layer when models A and B converged, and used kernel density analysis to visualize the distribution of layer-wise cosine similarities between the endpoints. The blue region represents the similarity after the first iteration, where we observed peak distributions close to −1 and 1, indicating a higher correlation between the models at this point. The orange region is more concentrated and centered around 0, suggesting that there was greater divergence between endpoints A and B at this stage.

We designed an ensemble learning experiment to assess the performance of the two models, Model A and Model B, on a test dataset. The objective was to compare the individual predictions from the models and evaluate the effectiveness of combining them. In each experiment batch, the predictions from Model A and Model B were combined using a weighted averaging method. Afterward, the softmax function was applied to these combined predictions to compute the final class probabilities.

The experimental results, as presented in

Table 4. Test results for ensemble model on CIFAR-10/100 and Vaihingen. The bold numbers indicate that the results from model ensemble (bagging) outperform those of individual models by more than 0.1%.

Cifar10 (ACC%)	SGD	TSE-A	TSE-B	A + B Bagging
resnet18	95.98	96.98	96.95	96.96
resnet50	96.2	97.02	97.12	97.27
wrn16-8	96.87	97.78	97.56	97.68
wrn28-2	95.78	96.41	96.55	96.59
wrn28-10	96.86	97.7	97.66	97.71
pyramidnet-110-48	95.85	96.92	96.62	96.93
pyramidnet-110-270	97.2	97.36	97.27	97.53
Cifar100 (ACC%)	SGD	TSE-A	TSE-B	A + B Bagging
resnet18	79.63	82.03	81.92	82.24
resnet50	78.74	83.45	83.31	83.90
wrn16-8	82.4	84.48	84.77	85.13
wrn28-2	77.7	79.92	80.73	81.16
wrn28-10	80.14	85.1	84.8	85.29
pyramidnet-110-48	78.35	82.98	82.83	83.20
pyramidnet-110-270	83.74	85.88	85.61	86.34
Vaihingen (mIoU%)	AdamW	TSE-A	TSE-B	A + B Bagging
UNetformer	82.7	85.07	84.9	85.08
LSKNet-S	85.1	85.4	85.47	85.56

, show the performance of both models individually, as well as when combined bagging. Specifically, we focused on integrating the logits (the raw output values before the softmax transformation) from both models after they made their final predictions. By combining these logits, we aimed to leverage the strengths of both models and improve the overall performance.

The results from the experiment demonstrate that combining models with different characteristics led to better predictive outcomes. The diversity between Model A and Model B allowed the ensemble method to capitalize on their complementary strengths. Additionally, our approach introduced an implicit form of orthogonal regularization, meaning that the models’ outputs, when integrated, exhibited reduced correlation. This regularization effect can help improve generalization and prevent overfitting, which further contributes to the enhanced performance of the ensemble model.

8.3.5. Results of Robustness to Label Noise

As demonstrated in previous work [9,14], both SAM (Sharpness-Aware Minimization) and ASAM (Adaptive Sharpness-Aware Minimization) have shown strong robustness to label noise in training data, making them effective in noisy environments. Building on these findings, we performed experiments using symmetric label noise at various noise levels: 0.2, 0.4, 0.6, and 0.8. This symmetric noise was introduced to mimic the effect of noisy or mislabeled data during the training process.

In our experiments, we evaluated the performance of our algorithm against that of standard SGD (Stochastic Gradient Descent), SAM, and ASAM at these different noise levels. The results indicate that, unlike SGD, which suffered from a significant performance drop as label noise increased, SAM and ASAM maintained relatively stable accuracies, even under noisy conditions. However, our proposed algorithm consistently outperformed both SAM and ASAM in terms of test accuracy across all noise levels, except for the 80% noise radio. This demonstrates the superior robustness of our approach, which effectively mitigated the impact of label noise, while boosting the overall model performance.

The detailed experimental results, which highlight the performance improvements of our algorithm(TSE) in various noisy conditions, are shown in

Table 5. Maximum test accuracies of ResNet-50 models trained on CIFAR-10 with label noise. The bold numbers indicate the best accuracy, signifying the best classification performance.

Noise Rate	SGD	SAM	ASAM	TSE
0	96.2	95.98	96.55	97.12
0.2	91.5	91.71	93.18	93.46
0.4	86.83	87.21	89.15	89.47
0.6	79.17	79.03	82.88	85.56
0.8	25.1	28.19	24.02	26.07

.

9. Discussion and Limitations

9.1. Failure Case Analysis

The main challenge with the algorithm arises from obtaining the model in the initial step. Factors influencing this include incorrectly set hyperparameters, such as $\rho$ and learning rate. If the initial models A and B are difficult to converge, this implies that the iterative training process may produce jumps. To avoid excessive consumption of computational resources, the training process can be further analyzed for biases through a visual analysis at each step, as shown in the Figure 10.

Visualizing the weights obtained at each step helps with error analysis. First, compare the weights obtained by the different optimizers and their performance metrics. In Figure 10, the linear interpolations among SGD, SAM, and ASAM exhibit higher loss barriers, suggesting that these optimization methods converge to distinct solutions. Meanwhile, there was no significant difference in performance, suggesting that the current training hyperparameter settings were reliable. Second, analyzing the performance connectivity between the current and previous steps helps determine whether the search iteration has introduced any biases, such as checking whether the distance regularization term produced the expected effect. Third, a comprehensive analysis of the connectivity between the solutions at each step can help determine if the process is converging to a flat region. In a good optimization process, the linear interpolation loss barriers between the models at later steps show a decreasing trend, indicating that the search direction is likely becoming flatter. Otherwise, it may be necessary to return and adjust the training hyperparameters.

9.2. Consumption Analysis

Our algorithm only adds a regularization term to each step of the test, resulting in no significant increase in memory consumption, apart from the overhead introduced by the regularization term. The regularization term requires storing the Fisher matrix of the network’s learnable weights.

The computational time of our algorithm is difficult to estimate directly. Therefore, we provide a table listing the number of forward and backward propagations during training. Assuming that the computational cost of forward and backward propagation for one epoch of stochastic gradient descent (SGD) is taken as 1 unit, and one full dataset pass consists of n epochs, we define the maximum number of iterations of our algorithm as I. The table below shows the worst-case computational cost of our algorithm:

The

Table 6. Consumption analysis.

SGD	SAM	TSE
n	$2n$	$(2\times 2n+2)I$

describes the worst-case scenario, where the number of training epochs per iteration is the same as in standalone training. To accelerate the process, an early stopping strategy can be applied during iterations, while setting a smaller number of epochs. The two additional passes added in each iteration are required to generate the Fisher matrix.

9.3. Stop Condition Analysis

The loss variance between two endpoints is a robust criterion for terminating iterations, ensuring consistent performance.

Table 7. Linear interpolation loss variance of iteratively output checkpoint models.

cifar10	Initial	Step 1	Step 2	Step 3	Step 4	Step 5
resnet18	0.793	0.767	0.009	0.003	0.002 *	0.0006
resnet50	0.74	0.789	0.362	0.12	0.08	0.07 *
wrn16-8	0.783	0.098	0.01	0.003 *	0.01	0.004
wrn28-2	0.77	0.67	0.11	0.04	0.04	0.04 *
wrn28-10	0.79	0.15	0.04	0.01	0.006	-
pyramidnet110-48	0.7	0.73	0.34	0.06	0.02 *	0.01
pyramidnet110-270	0.71	0.71	0.79	0.76	0.66	0.62 *
cifar100	initial	step 1	step 2	step 3	step 4	step 5
resnet18	1.588	1.433	0.232	0.007	0.01	0.01 *
resnet50	1.59	1.16	0.32	0.15	0.158 *	0.04
wrn16-8	1.65	0.32	0.09	0.04	0.01	0.005 *
wrn28-2	1.48	1.82	0.92	0.21	0.2	0.06 *
wrn28-10	1.71	0.37	0.05	0.01	0.006 *	-
pyramidnet110-48	1.45	1.57	0.97	0.25	0.09 *	0.05
pyramidnet110-270	1.51	1.53	1.65	1.43	1.04	0.9 *

’*’ denotes the best test result at the current step.

shows the loss variance of linear interpolation models between the endpoint models at each step. As iterations increased, this variance decreased monotonically. Optimal results typically occurred near the minimum variance step, except for ResNet-50 on CIFAR-100. Due to shared hyperparameters, PyramidNet-110-270 required more steps to reduce variance. We proposed a threshold of 0.1 as a stopping criterion. Additionally, visualizing the loss curves of the interpolation models at the conclusion of each iteration step provides significant insights, as shown in Figure 5 and Figure 6. Such visualizations enable the assessment of endpoint model convergence and facilitate a comparative analysis of the performance of the two endpoints, thereby informing decisions on whether to proceed with further iterations.

Note that we prefer to set a threshold where iterations are terminated once the variance falls below this threshold, thereby avoiding excessive iterations. One of the motivations behind our algorithm was to model the endpoints in search of flatter regions. Excessive iteration steps may cause the endpoints to gradually converge and eventually collapse into a single point.

9.4. Meta and Heuristic Learning

Our approach primarily emphasizes regularization and fine-tuning, aligning more closely with greedy algorithms within the heuristic optimization paradigm. However, the discussion of heuristic methods and meta-learning extends beyond the core scope of this paper. Nevertheless, our algorithm can be regarded as a foundation for further applications of heuristic algorithms, which, in turn, have the potential to enhance our method.

First, from the perspective of meta-learning, our method (TSE) lacks a memory mechanism. Meta-learning focuses on learning to learn, where the optimization process may carry more generalizable information about the loss landscape geometry [45,46,47]. In contrast, our TSE relies solely on the model state from the previous step for updates. A potential direction for improvement would be to incorporate multi-step historical information to enhance the optimization performance. Furthermore, integrating more comprehensive metrics could facilitate a more accurate assessment of whether the current search direction is appropriate.

Second, heuristic algorithms offer advantages in refining regularization terms and selecting search starting points. For instance, combinatorial optimization strategies such as evolutionary algorithms [48] and genetic algorithms [49] can optimize the selection of iteration starting points, while providing greater flexibility in adjusting the weights of regularization terms. The incorporation of these methods has the potential to further improve the efficiency and robustness of model optimization.

9.5. Future Directions

Our algorithm and visualization can provide a more intuitive way to search for flat regions. Our algorithm method offers an adaptive approach for flat region detection. This method follows a bounding strategy: it begins by assuming a sufficiently large flat region, and if the current loss landscape does not satisfy this assumption, the region is reduced and the search is repeated. Compared to the SAM cosine method, which sets a radius $\rho$, our approach is more purposeful. SAM lacks intuitive feedback to guide further hyperparameter tuning. As shown in Figure 4, the stop distance achieved for different network architectures indicates that SAM requires specific $\rho$ settings for each architecture, which involves extensive tuning and hyperparameter adjustments. Our method can be seen as a tool for intuitively understanding the flatness properties of the loss landscape in different network solution spaces.

Our method has led to two interesting observations, providing a foundation for further research. Assuming the results represent the flattest loss regions, future research may focus on the following:

1.

Model Architecture and Flat Region Size. As shown in Figure 4, the same network architecture corresponds to different flattest regions, depending on the dataset. Similarly, for the same dataset, different network architectures exhibit distinct flattest regions. It is possible that a normalization framework could unify this phenomenon. Furthermore, the size of the flattest region could guide the designing of model architectures.

2.

Euclidean Distance vs. Cosine Distance. As the Euclidean distance between two points decreases, the learned parameter vectors of the two models tend to become more orthogonal. This suggests that, although the Euclidean distance diminishes, model diversity increases. This observation is beneficial for better application of ensemble learning. When using ensemble learning to obtain a set of models, orthogonal regularization is often applied without considering the Euclidean distance between models. A potential issue arises: the generalization ability of an ensemble of models after orthogonal regularization is related to whether these models reside in the same loss basin. Our TSE algorithm provides a controllable tool to support further research in this area.

3.

Combining heuristic algorithms and meta-learning. A meta-learning algorithm memorizes gradient trajectory information, which is beneficial for designing better regularization and optimization iteration steps. Heuristic algorithms provide more diverse and efficient combinatorial forms, facilitating the selection of starting points and anchor points for the optimization iteration algorithm.

10. Conclusions

In this paper, we proposed an optimization strategy based on linear mode connectivity to adaptively adjust the hyperparameter $\rho$ in Sharpness-Aware Minimization (SAM). We innovatively employed a proof by contradiction to establish the theoretical relationship between the SAM hypersphere and connectivity. Furthermore, we introduced an iterative algorithm that progressively approximates flat regions. By analyzing the connectivity between A and B, our approach enables the adaptive adjustment of the SAM hypersphere radius $\rho$ without requiring additional manual tuning. Additionally, we proposed a distance regularization constraint and a corresponding TSE training process to guide A and B toward convergence in the same flat local minimum basin or adjacent low-loss basin, thereby enhancing optimization stability and generalization performance. Experimental results demonstrate that our method achieved superior convergence and generalization performance across multiple network architectures, significantly enhancing the model effectiveness.