Stochastic resetting mitigates latent gradient bias of SGD from label noise

Search processes are ubiquitous across various domains, spanning from systems in nature to applications in engineering. For instance, ligands exhibit search processes as they navigate toward target binding sites within proteins [21–23], and similarly, predators employ search strategies to locate their prey in the wild [24, 25]. In engineering, search processes are relevant to finding primary research studies [26], ranking web pages [27], and determining optimal hyperparameters for training algorithms [28]. Although diverse search strategies are employed depending on the problem at hand, they share a common goal: to identify an efficient search protocol. Efficiency is typically assessed by the time required to reach a target, referred to as the first passage time (FPT) in the context of random walk literature [29]. Numerous search strategies have been investigated to achieve this goal, including the Lévy strategies [30, 31], self-avoiding walks [32, 33], intermittent strategies [34], persistent random walks [35], and more [36]. One recent strategy that has garnered attention is stochastic resetting, with studies showcasing its ability to enhance search performance by revisiting previous places [2, 7, 10, 37]. In particular, these studies have demonstrated that stochastic restarts prevent a random searcher from wandering too far, thereby ensuring a finite mean time to find a target, whereas the mean time is infinite for a diffusive particle without resetting. Drawing from this concept, we introduce a resetting method for training DNNs and illustrate its effectiveness in addressing the noisy label problem.

While the accessibility of large datasets has propelled remarkable advancements in DNNs, the presence of noisy labels within these datasets often leads to erroneous model prediction [38]. Specifically, DNNs tend to overfit the entire corrupted training dataset by memorizing the wrong labels, which degenerates their generalization performance on a test dataset. Numerous studies have been conducted to address this overfitting phenomenon [39–44], and it has been revealed that DNNs initially learn the clean data (general patterns) during an early learning stage and then gradually memorize the corrupted data (task-specific patterns) [45–47]. The overfitting issue stemming from the memorization effect can be seen in figure 1(b), where the model’s accuracy in predicting the true labels of the data with noisy labels exhibits an inverted U-shaped curve as the model progressively memorizes the noise. Based on this understanding, the surprising effectiveness of the early-stopping method [48] in alleviating the memorization effect becomes evident; as such, various methods have been proposed to leverage this insight, including Co-teaching [49], SELFIE [50], early learning regularization (ELR) [51], and robust early learning [52]. Our proposed method also capitalizes on this insight by enabling the DNN to reset to a checkpoint, i.e. previously visited parameters during early learning stages, and implicitly serves as a regularization mechanism by indirectly affecting the SGD dynamics. Additionally, our theoretical analysis explores how label noise affects the performance of DNNs from the perspective of optimization strategies, such as those discussed in [47, 48]. We adopt a different approach based on statistical physics, assuming more practical settings, which offers novel insights and leads to the development of our method. In section 3, we provide a detailed analysis identifying a latent gradient bias of SGD due to label noise that causes the memorization effect, and how stochastic resetting can mitigate such negative effects.

When we apply the minibatch SGD to minimize the empirical risk $\mathcal{R}_{\mathcal{D}_\textrm{tr}}({\boldsymbol{\theta}})$ with respect to θ, the update rules of θ at each training iteration t can be represented by

$$\begin{align} \begin{aligned} \Delta {\boldsymbol{\theta}}_t & = -\frac{\eta}{B}\sum_{\left({\boldsymbol{x}}_i, {\boldsymbol{y}}_i\right)\in\mathcal{B}_t} {\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{L}_i\left({\boldsymbol{\theta}}_t\right) \\ & = -\frac{\eta}{N_\textrm{tr }}\sum_{\left({\boldsymbol{x}}_i, {\boldsymbol{y}}_i\right)\in\mathcal{D}_\textrm{tr}} {\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{L}_i\left({\boldsymbol{\theta}}_t\right) + \left( \frac{\eta}{N_\textrm{tr }}\sum_{\left({\boldsymbol{x}}_i, {\boldsymbol{y}}_i\right)\in\mathcal{D}_\textrm{tr}} {\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{L}_i\left({\boldsymbol{\theta}}_t\right) -\frac{\eta}{B}\sum_{\left({\boldsymbol{x}}_i, {\boldsymbol{y}}_i\right)\in\mathcal{B}_{t}} {\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{L}_i\left({\boldsymbol{\theta}}_t\right) \right), \end{aligned}\end{align} \tag{ 1 }$$

where ${\boldsymbol{\theta}}_t$ is θ at the t-th iteration, $\Delta {\boldsymbol{\theta}}_t \equiv {\boldsymbol{\theta}}_{t+1}-{\boldsymbol{\theta}}_t$, and $\mathcal{L}_i({\boldsymbol{\theta}}_t) \equiv \mathcal{L}({\boldsymbol{x}}_i, {\boldsymbol{y}}_i ;{\boldsymbol{\theta}}_t)$ for simplicity. Here, η > 0 is the learning rate and $\mathcal{B}_t$ is the minibatch of size B consisting of independent and identically distributed (i.i.d.) samples from $\mathcal{D}_\textrm{tr}$. While the first term on the right-hand-side (RHS) is deterministic for a given $\mathcal{D}_\textrm{tr }$, the second term on the RHS is stochastic due to the randomly sampled batch at each iteration. Thus, equation (1) can be rewritten as

$$\begin{align} \begin{aligned} \Delta {\boldsymbol{\theta}}_t = -{\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{R}_{\mathcal{D}_\textrm{tr}}\left({\boldsymbol{\theta}}_t\right) \eta + {\boldsymbol{\xi}}_t \sqrt{\eta}, \end{aligned}\end{align} \tag{ 2 }$$

where a random noise vector ${\boldsymbol{\xi}}_t \equiv \sqrt{\eta} \left( {\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{R}_{\mathcal{D}_\textrm{tr}}({\boldsymbol{\theta}}_t) - {\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{R}_{\mathcal{B}_{t}}({\boldsymbol{\theta}}_t) \right) \in \mathbb{R}^d$ satisfies $\langle {\boldsymbol{\xi}}_t \rangle_{\mathcal{D}_\textrm{tr}} = \boldsymbol{0}$ and $\langle {\boldsymbol{\xi}}_t {\boldsymbol{\xi}}_s^\textrm{T} \rangle_{\mathcal{D}_\textrm{tr}} = 2 \mathsf{D}({\boldsymbol{\theta}}_t) \delta_{ts}$ with $\mathsf{D}({\boldsymbol{\theta}}_t) \equiv \eta \sf{\Sigma}({\boldsymbol{\theta}}_t) /(2B)$, where δ_ij denotes the Kronecker delta (see details in the Supplementary Materials (SM) and references [53–55]). In terms of Langevin dynamics, $\mathcal{R}_{\mathcal{D}_\textrm{tr}}({\boldsymbol{\theta}})$ and $\mathsf{D}({\boldsymbol{\theta}})$ correspond to the potential and diffusion matrix, respectively, where the former generates the deterministic long-term trend called drift and the latter determines the level of stochasticity of the system [56]. As a result, the SGD dynamics of ${\boldsymbol{\theta}}_t$ can be understood by the Langevin dynamics of a d-dimensional particle diffusing with drift $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{R}_{\mathcal{D}_\textrm{tr}}({\boldsymbol{\theta}}_t)$ and diffusion matrix $\mathsf{D}({\boldsymbol{\theta}}_t)$.

For a corrupted dataset $\tilde{\mathcal{D}}_\textrm{tr}$, the equation of SGD dynamics remains analogous to equation (2) when we substitute ${\mathcal{D}}_\textrm{tr}$ with $\tilde{\mathcal{D}}_\textrm{tr}$. Then we can divide the drift vector $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{R}_{\tilde{\mathcal{D}}_\textrm{tr}}({\boldsymbol{\theta}}_t)$ into two components, one originating from $\tilde{\mathcal{D}}_\textrm{tr}^c$ and the other from $\tilde{\mathcal{D}}_\textrm{tr}^w$ as follows (figure 2(a)):

$$\begin{align} \begin{aligned} \Delta {\boldsymbol{\theta}}_t = -\left[ {\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^c}\left({\boldsymbol{\theta}}_t\right) + {\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^w}\left({\boldsymbol{\theta}}_t\right) \right] \eta + {\boldsymbol{\xi}}_t \sqrt{\eta}, \end{aligned}\end{align} \tag{ 3 }$$

with the gradients from the correct part ${\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^c}({\boldsymbol{\theta}}_t) \equiv (1-\tau){\boldsymbol{\nabla}}_{\boldsymbol{\theta}} {\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^c}({\boldsymbol{\theta}}_t)$ and the gradients from the wrong part ${\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^w}({\boldsymbol{\theta}}_t) \equiv \tau{\boldsymbol{\nabla}}_{\boldsymbol{\theta}} {\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^w}({\boldsymbol{\theta}}_t)$. We refer to the drift from the wrong part $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^w}({\boldsymbol{\theta}}_t)$ as the latent gradient bias by label noise. Note that $\langle {\boldsymbol{\nabla}}_{\boldsymbol{\theta}}{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^c}({\boldsymbol{\theta}}_t) \rangle_{\mathcal{D}} = \langle {\boldsymbol{\nabla}}_{\boldsymbol{\theta}} {\mathcal{R}}_{{\mathcal{D}}_\textrm{tr}}({\boldsymbol{\theta}}_t) \rangle_{\mathcal{D}}$, implying that $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}}\hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^c}({\boldsymbol{\theta}}_t)$ reflects the gradients toward the true optimum, while $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}}\hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^w}({\boldsymbol{\theta}}_t)$ reflects the gradients toward the false optimum by memorizing the noisy labels. Additionally, we observe that $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}}\hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^c}({\boldsymbol{\theta}}_t)$ and $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}}\hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^w}({\boldsymbol{\theta}}_t)$ are orthogonal to each other (figures 2(b) and S.1 in the SM), leading to $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{R}_{\tilde{\mathcal{D}}_\textrm{tr}}({\boldsymbol{\theta}}_t)$ being represented by the sum of two orthogonal vectors. Thus, $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{R}_{\tilde{\mathcal{D}}_\textrm{tr}}({\boldsymbol{\theta}}_t)$ becomes more correlated with $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}}\hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^w}({\boldsymbol{\theta}}_t)$ as the noise rate τ increases. Figures 2(b) and (c) illustrate that $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}}\hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^w}({\boldsymbol{\theta}}_t)$ becomes increasingly dominant so that the drift gradually tilts toward wrong directions as τ increases, where $\cos \phi_{tw}$ denotes the cosine similarity between $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}}\hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^c}({\boldsymbol{\theta}}_t)$ and $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}}\hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^w}({\boldsymbol{\theta}}_t)$, and $\| \cdot \|$ denotes the Euclidean norm of a vector. This gradually tilting trend toward a wrong direction can also be observed with respect to iteration t, implying that $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}}\hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^w}({\boldsymbol{\theta}}_t)$ becomes increasingly dominant as the learning process progresses beyond an early learning phase [51]. Therefore, in the presence of noisy labels, we can see that the latent gradient bias emerges and hinders the search for the optimal parameters ${\boldsymbol{\theta}}^*$.

Zoom In Zoom Out Reset image size

We note that a similar analysis of the SGD dynamics using statistical physics was previously performed in [47]. While both studies observed an increasing trend in the effect of latent gradient bias during the learning process, their findings on the cosine similarity between drift components, $\cos \phi_{cw}$, differ from ours. To address this discrepancy, we conducted additional experiments and identified the contributing factors, as detailed in section D.2 of the SM.

We now describe how the stochastic resetting method can be integrated into SGD. Based on this, we establish the specific premises of this work. Let ${\boldsymbol{\theta}}_c$ be a reset checkpoint and r be the reset probability at each iteration t, where a checkpoint refers to previously visited model parameters during training. By incorporating the resetting method, equation (2) for $\tilde{\mathcal{D}}_\textrm{tr}$ can be expressed as

$$\begin{align} \begin{aligned} {\boldsymbol{\theta}}_{t+1} = \begin{cases} {\boldsymbol{\theta}}_c, &\text{with probability r}, \\ {\boldsymbol{\theta}}_{t} -{\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{R}_{\tilde{\mathcal{D}}_\textrm{tr}}\left({\boldsymbol{\theta}}_t\right) \eta + {\boldsymbol{\xi}}_t \sqrt{\eta}, &\text{otherwise, equation (3)}. \end{cases} \end{aligned}\end{align} \tag{ 4 }$$

Below we provide the pseudo-code for SGD with stochastic resetting, algorithm 1.

Algorithm 1. Stochastic resetting.
require Corrupted training set $\tilde{\mathcal{D}}_\textrm{tr}$, validation set $\mathcal{D}_\textrm{val}$, reset probability r, threshold $\mathcal{T}$.
1: Initialize ${\boldsymbol{\theta}}_0$ and set t = 0, ${\boldsymbol{\theta}}_c = \text{None}$, and ${\boldsymbol{\theta}}_\textrm{best} = \text{None}$
2: for t = 0 to T do
3:   Update ${\boldsymbol{\theta}}_{t} \leftarrow {\boldsymbol{\theta}}_t -\frac{\eta}{B}\sum_{({\boldsymbol{x}}_i, {\boldsymbol{y}}_i)\in\mathcal{B}_t} {\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \mathcal{L}_i({\boldsymbol{\theta}}_t)$ where $\mathcal{B}_t$ is a randomly sampled batch from $\tilde{\mathcal{D}}_\textrm{tr}$
4:   if ${\boldsymbol{\theta}}_c \neq \text{None}$ and $rand(0, 1) \lt r$ then
5:     Restart ${\boldsymbol{\theta}}_{t} \leftarrow {\boldsymbol{\theta}}_c$
6:   end if
7:   ${\boldsymbol{\theta}}_\textrm{best} \leftarrow \text{Valid}({\boldsymbol{\theta}}_{t}, \mathcal{D}_\textrm{val})$ where $\text{Valid}({\boldsymbol{\theta}}_{t}, \mathcal{D}_\textrm{val})$ checks whether $\mathcal{R}_{\mathcal{D}_\textrm{val}} ({\boldsymbol{\theta}}_t)$ is the minimum.
8:   if ${\boldsymbol{\theta}}_\textrm{best}$ remains unchanged for $\mathcal{T}$ iterations or ${\boldsymbol{\theta}}_t = {\boldsymbol{\theta}}_\textrm{best}$ then
9:     Set the checkpoint ${\boldsymbol{\theta}}_c \leftarrow {\boldsymbol{\theta}}_\textrm{best}$
10:   end if
11: end for

The SGD dynamics with stochastic resetting involves two processes: resetting from a checkpoint ${\boldsymbol{\theta}}_c$ with probability r (top of equation (4)), and maintaining the SGD dynamics with probability $1-r$ (bottom of equation (4)). Note that equation (4) shares the same form as the (overdamped) Langevin equation with Poissonian reset [3], and also that training DNNs to find optimal parameters can be likened to a search process for an unknown target. These parallels imply that similar advantages of stochastic resetting may arise in the training process of DNNs as in the random search process of Langevin dynamics. To examine this hypothesis, we first explore the beneficial mechanism of stochastic resetting in the search process with a simplified case.

The search efficiency of a random search process is typically quantified by the mean first passage time (MFPT) [29], which represents the average time to find a target. It has been well-established that incorporating stochastic resetting can significantly reduce the MFPT in various complex scenarios, including high-dimensional spaces [1], various confining potentials [7, 57], and a searcher with momentum [6, 58], among others [2]. For a simplified case, let us consider a random searcher in one dimension with diffusion coefficient D, drift v, and reset rate γ ($\equiv r/\Delta t$ with a time interval Δt between steps), and assume that the searcher starts at the origin (reset point). Then the MFPT for a target located at L ($\gt 0$) can be expressed by

$$\begin{align} \begin{aligned} \langle T\left(\gamma\right) \rangle = \frac{1}{\gamma} \left[ e^{\frac{L}{2D} \left(\sqrt{v^2 + 4D\gamma} - v \right)} - 1\right], \end{aligned}\end{align} \tag{ 5 }$$

where $\langle T(\gamma) \rangle$ denotes the MFPT with the reset rate γ (see the derivation in section B of the SM). Examining equation (5) provides several insights into the effects of stochastic resetting. When the random searcher either normally diffuses or drifts away from the target ($v \unicode{x2A7D} 0$ ), it is straightforward that $\langle T(\gamma) \rangle$ diverges as $\gamma \rightarrow 0$ but becomes finite for any γ > 0. Conversely, when the searcher drifts toward the target (v > 0), while $\langle T(\gamma) \rangle$ is finite as $L/v$ without resetting, introducing stochastic resetting can significantly reduce $\langle T(\gamma) \rangle$ within a certain range of γ, provided the following condition is met:

$$\begin{align} \begin{aligned} \textrm{Pe} \equiv \frac{L v}{2D} \unicode{x2A7D} 1. \end{aligned}\end{align} \tag{ 6 }$$

Here, $\textrm{Pe}$ is known as the Péclet number, which quantifies the ratio between drift and diffusive transport rates, and the beneficial condition ($\textrm{Pe} \unicode{x2A7D} 1$) can be identified by verifying where $[d\langle T(\gamma) \rangle/d\gamma]|_{\gamma \rightarrow 0} \lt 0$. Figure 3 illustrates how the behavior of $\langle T(\gamma) \rangle$ evolves with varying $\textrm{Pe}$: as $\textrm{Pe}$ decreases, the initially monotonically increasing curve gradually transforms into a U-shaped curve, achieving a minimum $\langle T(\gamma) \rangle$ at the optimal reset rate $\gamma^* \gt 0$. These findings indicate that resetting is advantageous for a target search when the stochasticity (D) is sufficiently larger than the drift toward a target (v). Furthermore, we note that both the optimal reset rate $\gamma^*$ and the improvement ratio $\langle T(0) \rangle/\langle T(\gamma^*) \rangle$ increase as $\textrm{Pe}$ decreases (figure 3), indicating that the benefits of resetting grow as D increases and v decreases.

Zoom In Zoom Out Reset image size

According to the above observations, the beneficial properties of resetting in a random search process can be summarized as follows:

The advantageous mechanism of resetting in a random search process is to suppress trajectories that move away from the target, thus increasing the chances of finding it. Drawing a parallel between the random search process and the training procedure of DNNs via SGD, we hypothesize this mechanism is also applicable to the noisy label problem in DNNs. Specifically, in supervised learning with noisy labels, the stochasticity of the SGD dynamics increases as batch size B decreases, and the drift component toward a target $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \hat{\mathcal{R}}_{\tilde{\mathcal{D}}^c_\textrm{tr}}({\boldsymbol{\theta}}_t)$ weakens (i.e. the latent gradient bias strengthens) as the noise rate τ increases. Therefore, the boxed statement suggests that the resetting strategy would be beneficial for searching for optimal parameters in cases with a small batch size and large noise rate in the presence of noisy labels. In section 4, we perform several experiments and empirically show that stochastic resetting enhances generalization performance.

To introduce the resetting strategy in DNN training, we first explore which checkpoint is suitable to reset to in order to find optimal parameters. A straightforward choice is to select the parameters at the overfitting iteration t_m. Here, the overfitting iteration t_m refers to an iteration where the validation loss ceases to decrease and begins to increase due to the memorization effect [inset of figure 1(b)]. The checkpoint at t_m, denoted by ${\boldsymbol{\theta}}_m$, has the minimum validation loss during training when the double descent phenomenon does not occur [63], and is typically employed as an early-stopping point. Instead of early stopping and considering ${\boldsymbol{\theta}}_m$ as the final model, we utilize ${\boldsymbol{\theta}}_m$ as the checkpoint ${\boldsymbol{\theta}}_c$ to reset to (algorithm 1), leading to significantly improved results (figure 4(a)). Here, ${\boldsymbol{\theta}}_c$ is initially set to ${\boldsymbol{\theta}}_m$ and adaptively changes to the parameter at a newly found minimum validation loss during training. As can be seen in figure 4(a), resetting suppresses the trajectory of θ to be near ${\boldsymbol{\theta}}_c$, which successfully prevents memorizing the noisy labels and increases the chance to find more appropriate parameters. It is important to note that the reset probability r controls the degree of suppression and the results at r = 0 and r = 1 are almost the same due to the overfitting phenomenon (the case of r = 1 corresponds to the early-stopping method). Therefore, the resulting RDVLoss curve for r should be U-shaped, indicating that an optimal r exists to optimize the performance.

Zoom In Zoom Out Reset image size

While we simply select ${\boldsymbol{\theta}}_m$ as the initial ${\boldsymbol{\theta}}_c$ and adaptively update it, one may ask what effect the choice of ${\boldsymbol{\theta}}_c$ has. To check this, we experiment with a fixed checkpoint both earlier and later than ${\boldsymbol{\theta}}_m$. For earlier checkpoints (left panel in figure 4(b)), the improvement over resetting from ${\boldsymbol{\theta}}_m$ slightly decreases and the value of the optimal r gets smaller as the checkpoint gets earlier. In contrast, for later checkpoints (right panel in figure 4(b)), the improvement over resetting from ${\boldsymbol{\theta}}_m$ significantly decreases as the checkpoint gets later. These results support our understanding of the beneficial mechanism of resetting in increasing the chance of finding better parameters, because the chance would decrease as the model memorizes more noise. Therefore, we can conclude that resetting in early learning stages is a good choice: the more memorization occurs, the smaller the improvement.

We additionally experiment to verify how the effect of resetting changes with the distance between the (adaptive) checkpoint ${\boldsymbol{\theta}}_c$ at the minimum validation loss and a perturbed checkpoint ${\boldsymbol{\theta}}_{c, \epsilon}$. Here, we set the perturbed checkpoint by adding the perturbation $\epsilon \hat{\boldsymbol{n}}$ into ${\boldsymbol{\theta}}_c$ with varying the perturbation magnitude ε, where $\hat{\boldsymbol{n}} \equiv {\boldsymbol{n}}/\|{\boldsymbol{n}}\| \in \mathbb{R}^d$ is a random unit vector with a standard normal random vector n. As shown in figure 4(c), it is observed that the benefits of resetting decrease as the distance between the checkpoint to reset to and ${\boldsymbol{\theta}}_m$ increases. This result also supports that while resetting can improve the generalization performance, the choice of checkpoint to reset to can affect the performance, and that the minimum validation loss point is a good choice.

As the statement highlighted in section 3.2 clarifies, it has been proven in the statistical physics field that the resetting strategy can improve search efficiency as the stochasticity becomes larger than the drift component toward a target. In this section, we verify whether this statement is also valid in the training of DNNs and show under what circumstances resetting is more effective than not resetting.

It is first important to note that the stochasticity and the drift toward a target, i.e. $-{\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^c}$, can be controlled by the batch size B and the noise rate τ, respectively, as illustrated in section 3.1. Particularly, $\mathsf{D}({\boldsymbol{\theta}}_t) \propto 1/B$ and ${\boldsymbol{\nabla}}_{\boldsymbol{\theta}} \hat{\mathcal{R}}_{\tilde{\mathcal{D}}_\textrm{tr}^c}({\boldsymbol{\theta}}_t) \propto 1-\tau$, meaning that the stochasticity increases and the drift toward a target decreases as B decreases and τ increases, respectively. We quantitatively examine the RDVLoss and the RDTAcc. values with varying B and τ with respect to the reset probability r. Remarkably, the improvements of RDVLoss and RDTAcc. with resetting become more significant as B decreases (figure 5(a)) and τ increases (figure 5(b)). These observations strongly support our claim that stochastic resetting offers more benefits as the stochasticity increases and the drift toward a target decreases.

Zoom In Zoom Out Reset image size

Moreover, we expect that the optimal reset probability $r^*$ decreases as B decreases and τ increases, but this can only be verified qualitatively because the fluctuation of the results makes it difficult to identify $r^*$.

Until now, we have leveraged the memorization effect in our algorithm by utilizing the parameters at the minimum validation loss as the checkpoint for resetting the entire network, a process referred to as full resetting. However, several studies have highlighted that different layers within a DNN exhibit varied learning behaviors, leading to distinct levels of overfitting across these layers [64, 65]. A prevailing explanation for this phenomenon suggests that gradients tend to weaken as they propagate from the latter layers (closer to the output layer) to the former layers (closer to the input layer).

Here, we experiment on which layers, former or latter, play a more dominant role in improving performance with the resetting method. For this, we introduce partial resetting, which involves resetting only one section of the network layers rather than the entire network, while the remaining section of layers continues to follow the standard SGD update rule without resetting. We divide the VCNN structure into former and latter sections, comprising convolutional and linear layers, respectively, and apply partial resetting to one section. Interestingly, our experiments reveal that partial resetting of the latter section can further enhance generalization performance compared to full resetting, whereas partial resetting of the former section does not yield improvements over the case with no resetting (r = 0) (figure 6).

Zoom In Zoom Out Reset image size

Moreover, even when we freeze the latter section by setting r = 1, partial resetting of the latter section still achieves significant improvement. We attribute these findings to a well-established observation: the former layers of CNNs tend to learn general features, while the latter layers tend to specialize in capturing specific features [64, 66–69]. In other words, the latter section composed of linear layers exhibits strong memorization of the corrupted data, leading to improved performances even when we freeze the latter section (r = 1), whereas the former section composed of convolutional layers focuses on learning general features, leading to no improvements even with resetting.

It is essential to note that although our ablation study suggests the effectiveness of partial resetting, our findings do not imply that partial resetting of the latter section always enhances generalization performance compared to full resetting. The extent to which each layer overfits the corrupted data depends on multiple factors, such as the network structure and the choice of loss function. Thus, determining the most effective section of the network to reset also hinges on the specific context. Future research investigating these points would be intriguing and valuable.

In Setting 2, we investigate the impact of the stochastic resetting strategy on the performance of benchmark datasets, CIFAR-10 and CIFAR-100, under symmetric noise corruption using various methods. Table 1 compares the best results without resetting (No) and with resetting (Reset) at the reset probability r = 0.001. In the table, CE denotes cross-entropy loss, PartRestart denotes cross-entropy loss with partial resetting of only the last linear layer and the last two blocks of ResNet-34 (section 4.3), MAE denotes robust mean absolute error [70], GCE denotes generalized cross-entropy [71], SL denotes symmetric cross-entropy loss [72], ELR denotes early-learning regularization [51], and SOP+ denotes sparse over-parameterization with consistency regularization and class-balance regularization [73]. ELR and SOP+ are representative methods to robustly train DNNs with an additional regularization term to prevent overfitting to corrupted data. However, these methods require additional hyperparameters to be fine-tuned depending on the loss landscape (e.g. dataset, model architecture), and in practical scenarios, it is often costly to find the optimal settings of these methods. In this section, we consider both optimal and non-optimal hyperparameter settings to take such practical situations into account, where we use an asterisk (*) to denote a non-optimal hyperparameter setting (i.e. ELR*, SOP+*). We provide additional details about the hyperparameters for each method in section C.2 of the SM.

Table 1. Test accuracies (%) on test datasets with different methods. We compare the performance without resetting (No) and with resetting (Reset) at r = 0.001. Results are presented as the average and the standard deviation. The best results are indicated in bold with statistical significance.

Dataset	Method	Noise rate τ = 0.2		Noise rate τ = 0.4		Noise rate τ = 0.6
		No	Reset	No	Reset	No	Reset
CIFAR-10	CE	$84.8\pm0.4$	$\boldsymbol{90.0\pm0.5 ^{\scriptscriptstyle ***}}$	$80.8\pm0.7$	$\boldsymbol{86.3\pm0.7 ^{\scriptscriptstyle ***}}$	$72.9\pm0.7$	$\boldsymbol{79.5\pm0.9^{\scriptscriptstyle ***}}$
	Part	—	$\boldsymbol{88.5\pm0.4^{\scriptscriptstyle ***}}$	—	$\boldsymbol{83.7\pm0.9^{\scriptscriptstyle ***}}$	—	$\boldsymbol{75.1\pm0.4^{\scriptscriptstyle ***}}$
	MAE	$91.0\pm0.2$	$91.0\pm0.2$	$85.6\pm3.3$	$\boldsymbol{85.9\pm3.3}$	$67.1\pm6.8$	$\boldsymbol{67.3\pm6.7}$
	GCE	$90.6\pm0.1$	$\boldsymbol{91.0\pm0.2^{\scriptscriptstyle *}}$	$85.3\pm0.4$	$\boldsymbol{87.1\pm0.3^{\scriptscriptstyle ***}}$	$76.2\pm0.5$	$\boldsymbol{79.1\pm0.8^{\scriptscriptstyle ***}}$
	SL	$91.3\pm0.3$	$\boldsymbol{91.4\pm0.1}$	$86.6\pm0.2$	$\boldsymbol{87.9\pm0.2^{\scriptscriptstyle ***}}$	$80.1\pm0.5$	$\boldsymbol{81.6\pm0.6^{\scriptscriptstyle **}}$
	ELR*	$91.2\pm0.1$	$\mathbf{91.3\pm0.2}$	$88.8\pm0.2$	$\mathbf{88.9\pm0.2}$	$84.6\pm0.5$	$84.6\pm0.5$
	SOP+*	${94.1\pm0.2}$	$94.1\pm0.2$	$89.9\pm0.3$	$89.9\pm0.3$	$85.0\pm0.3$	$\mathbf{85.1\pm0.3}$
CIFAR-100	CE	$62.7 \pm 5.6$	$\boldsymbol{64.5 \pm 1.5}$	$45.6 \pm 2.3$	$\boldsymbol{56.9 \pm 3.0^{\scriptscriptstyle ***}}$	$32.9 \pm 1.6$	$\boldsymbol{44.1 \pm 3.1^{\scriptscriptstyle ***}}$
	Part	—	$\boldsymbol{64.0\pm0.9}$	—	$\boldsymbol{55.5\pm2.1^{\scriptscriptstyle ***}}$	—	$\boldsymbol{43.5\pm1.5^{\scriptscriptstyle ***}}$
	MAE	$19.9 \pm 2.8$	$19.9 \pm 2.8$	$\boldsymbol{11.0 \pm 3.8}$	$10.8 \pm 3.7$	$6.7 \pm 1.2$	$\boldsymbol{6.8 \pm 1.3}$
	GCE	$68.3 \pm 0.4$	$\boldsymbol{69.3 \pm 0.2^{\scriptscriptstyle **}}$	$61.2 \pm 0.6$	$\boldsymbol{63.2 \pm 0.3^{\scriptscriptstyle ***}}$	$50.1 \pm 0.6$	$\boldsymbol{53.1 \pm 0.8^{\scriptscriptstyle ***}}$
	SL	$66.8\pm0.5$	$\boldsymbol{68.6\pm0.6^{\scriptscriptstyle ***}}$	$60.4\pm0.5$	$\boldsymbol{63.1\pm0.7^{\scriptscriptstyle ***}}$	$50.4\pm0.8$	$\boldsymbol{54.1\pm0.8^{\scriptscriptstyle ***}}$
	ELR*	$67.4\pm0.3$	$\mathbf{70.4\pm0.3^{\scriptscriptstyle ***}}$	$55.1\pm0.6$	$\mathbf{64.1\pm0.6^{\scriptscriptstyle ***}}$	$45.9\pm0.7$	$\mathbf{54.0\pm1.0^{\scriptscriptstyle ***}}$
	SOP+*	${72.1\pm0.3}$	$72.1\pm0.3$	$59.4\pm0.5$	$\mathbf{65.7\pm0.9^{\scriptscriptstyle ***}}$	$48.4\pm1.9$	$\mathbf{54.8\pm2.3^{\scriptscriptstyle **}}$

Remarkably, in all cases examined, our resetting method consistently achieves either at least equivalent or higher test accuracies compared to the baseline approach involving no resetting (table 1). Results show that the extent of improvement becomes more pronounced as the noise rate increases, which supports our claim that resetting becomes more advantageous with higher noise rates. Furthermore, while the PartRestart method also obtains improved performance, it does not surpass the benefits of full resetting. We conjecture that the network structure may influence the extent of the additional improvements obtained, as the memorization effect in different layers can vary depending on the network architecture [74]. Minimal improvements are observed in the MAE results for both datasets; this is primarily because the MAE convergence is too slow to identify a suitable checkpoint for resetting, consequently resulting in few instances of resetting in many trials. For ELR and SOP+, no significant improvements are found in the performance when stochastic resetting is used with the known optimal hyperparameter settings, as shown in table S.4 in the SM. However, for ELR* and SOP+*, incorporating the stochastic resetting method leads to either at least equivalent performance or significant improvements compared to no resetting. This can be explained by the design of ELR and SOP+ that provide regularization terms to prevent memorization, akin to reducing the effect of the latent gradient bias. These results indicate that stochastic resetting is compatible with well-established methods, and can also provide at least equal performance compared to the methods without resetting.

We likewise demonstrate the effectiveness of the stochastic resetting method in asymmetric (i.e. class-dependent) noise scenarios (table S.5 in the SM), and we also verify the improvement of the validation loss when using our stochastic resetting method (table S.6 in the SM).

Finally, we test our stochastic resetting method on real-world noisy datasets, namely CIFAR-10N/100N (Setting 3). There are five different noise types for CIFAR-10N, namely Rand1, Rand2, Rand3, Aggregate, and Worst with a noise rate of 9.03%, 17.23%, 18.12%, 17.64%, and 40.21%, respectively, and a single noise type for CIFAR-100N with a noise rate of 40.20%. For CIFAR-10N, we selected Rand1, Aggregate, and Worst noise types to account for various real-world noise rates. We further evaluate our method on ANIMAL-10N with the default setting of Setting 3 using cross-entropy loss. For the ANIMAL-10N dataset, there is only a single noise type while the ground-truth labels remain unknown. The noise rate τ was estimated as $\tau \sim 0.08$ using cross-validation with a grid search and $\tau \sim 0.06$ based on human inspection, respectively.

Table 2 presents the performance results without resetting (No) and with resetting (Reset) at the reset probability r = 0.001 for CIFAR-10N/100N. Similarly, the results for ANIMAL-10N are shown in table 3. We provide additional details about the hyperparameters in section C.2 of the SM. For the case of CIFAR-10N, the stochastic resetting method consistently achieves either at least equivalent or higher test accuracies compared to the baseline without resetting, similar to the results in section 4.4 (table 1). While the experiments on CIFAR-10N show some cases with minimal improvements from the stochastic resetting method, the results on CIFAR-100N and ANIMAL-10N provide strong evidence of the advantages of applying stochastic resetting in practical scenarios. These results demonstrate that the stochastic resetting method can provide significant improvements while acting as a safeguard to maintain baseline performance.

Table 2. Test accuracies (%) on test datasets with real-world datasets CIFAR-10N/100N. We compare the performance without resetting (No) and with resetting (Reset) at r = 0.001. Results are presented as the average and the standard deviation. The best results are indicated in bold with statistical significance.

Method	CIFAR-10N		CIFAR-10N		CIFAR-10N		CIFAR-100N
	Random 1 ($\tau \sim 0.09$)		Aggregate ($\tau \sim 0.17$)		Worst ($\tau \sim 0.4$)		Noisy ($\tau \sim 0.4$)
	No	Reset	No	Reset	No	Reset	No	Reset
CE	$82.8\pm0.5$	$\boldsymbol{87.8\pm0.7^{\scriptscriptstyle ***}}$	$86.1\pm0.6$	$\boldsymbol{90.2\pm0.3^{\scriptscriptstyle ***}}$	$73.8\pm1.9$	$\boldsymbol{79.7\pm0.3^{\scriptscriptstyle ***}}$	$46.7\pm1.6$	$\boldsymbol{56.0\pm1.3^{\scriptscriptstyle ***}}$
MAE	$90.9\pm0.3$	$90.9\pm0.2$	$90.0\pm0.5$	$90.0\pm0.5$	$60.2\pm3.6$	$60.2\pm3.6$	$4.0\pm1.3$	$4.0\pm1.3$
SL	$91.7\pm0.2$	$\boldsymbol{91.8\pm0.2}$	$89.6\pm0.2$	$\boldsymbol{89.9\pm0.4}$	$80.6\pm0.5$	$\boldsymbol{81.3\pm0.8}$	$52.3\pm0.5$	$\boldsymbol{56.5\pm0.6^{\scriptscriptstyle ***}}$
GCE	$91.7\pm0.2$	$\boldsymbol{91.9\pm0.2}$	$89.4\pm0.4$	$\boldsymbol{89.8\pm0.7}$	$78.1\pm0.9$	$\boldsymbol{82.3\pm0.5^{\scriptscriptstyle ***}}$	$55.8\pm0.5$	$\boldsymbol{58.0\pm1.2^{\scriptscriptstyle ***}}$
ELR*	${89.5\pm0.3}$	${89.5\pm0.3}$	${91.3\pm0.2}$	${91.3\pm0.2}$	${81.9\pm0.4}$	${81.9\pm0.5}$	$56.9\pm0.4$	$\boldsymbol{60.5\pm0.5^{\scriptscriptstyle ***}}$
SOP+*	$91.2\pm0.2$	${91.2\pm0.2}$	$\boldsymbol{93.2\pm0.1}$	${93.0\pm0.1}$	$\boldsymbol{82.4\pm0.5}$	${82.3\pm0.7}$	$57.0\pm0.4$	$\boldsymbol{61.2\pm0.9^{\scriptscriptstyle ***}}$
ViT-T$_\textrm{rd}$	$63.2\pm0.9$	$\boldsymbol{65.7\pm0.7^{\scriptscriptstyle ***}}$	${61.0\pm0.9}$	$\boldsymbol{63.6\pm0.3^{\scriptscriptstyle ***}}$	${55.4\pm0.2}$	$\boldsymbol{57.2\pm0.2^{\scriptscriptstyle **}}$	$28.6\pm0.5$	$\boldsymbol{32.8\pm0.5^{\scriptscriptstyle ***}}$
ViT-T$_\textrm{pt}$	$92.6\pm0.3$	$\boldsymbol{93.6\pm0.6^{\scriptscriptstyle *}}$	${91.8\pm0.5}$	$\boldsymbol{93.0\pm0.8^{\scriptscriptstyle **}}$	${85.9\pm1.4}$	$\boldsymbol{87.4\pm0.7^{\scriptscriptstyle *}}$	$63.8\pm1.1$	$\boldsymbol{68.6\pm0.5^{\scriptscriptstyle **}}$

Table 3. Test accuracies (%) on test datasets with real-world datasets ANIMAL-10N. We compare the performance without resetting (No) and with resetting (Reset) at r = 0.001.

Dataset	No	Reset
ANIMAL-10N	$80.6\pm0.6$	$\boldsymbol{85.1\pm0.4 ^{\scriptscriptstyle ***}}$

Furthermore, to assess the applicability of our method beyond CNN-based approaches, we evaluate it on the Vision Transformer (ViT) [75]. As shown in table 2, the resetting method consistently improves performance on ViT, denoted by ViT-T_rd and ViT-T_pt, which correspond to ViT-Tiny models [76] trained from randomly initialized and pre-trained weights, respectively. Previous studies have reported that ViTs without pre-training are more prone to overfitting, particularly in the presence of label noise [77, 78], which we also observe in table 1. Interestingly, regardless of initialization, the resetting method results in significant performance improvements, highlighting its potential for a broader range of recent architectures and scenarios.