Shortcuts

# CosineAnnealingLR¶

class mmengine.optim.CosineAnnealingLR(optimizer, *args, **kwargs)[source]

Set the learning rate of each parameter group using a cosine annealing schedule, where $$\eta_{max}$$ is set to the initial value and $$T_{cur}$$ is the number of epochs since the last restart in SGDR:

\begin{split}\begin{aligned} \eta_t & = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right), & T_{cur} \neq (2k+1)T_{max}; \\ \eta_{t+1} & = \eta_{t} + \frac{1}{2}(\eta_{max} - \eta_{min}) \left(1 - \cos\left(\frac{1}{T_{max}}\pi\right)\right), & T_{cur} = (2k+1)T_{max}. \end{aligned}\end{split}

Notice that because the schedule is defined recursively, the learning rate can be simultaneously modified outside this scheduler by other operators. If the learning rate is set solely by this scheduler, the learning rate at each step becomes:

$\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right)$

It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts. Note that this only implements the cosine annealing part of SGDR, and not the restarts.

Parameters
• optimizer (Optimizer or OptimWrapper) – Wrapped optimizer.

• T_max (int) – Maximum number of iterations.

• eta_min (float) – Minimum learning rate. Defaults to 0.

• begin (int) – Step at which to start updating the learning rate. Defaults to 0.

• end (int) – Step at which to stop updating the learning rate. Defaults to INF.

• last_step (int) – The index of last step. Used for resume without state dict. Defaults to -1.

• by_epoch (bool) – Whether the scheduled learning rate is updated by epochs. Defaults to True.

• verbose (bool) – Whether to print the learning rate for each update. Defaults to False.

© Copyright 2022, mmengine contributors. Revision 13484aae.

Built with Sphinx using a theme provided by Read the Docs.
Read the Docs v: v0.2.0
Versions
latest
stable
v0.5.0
v0.4.0
v0.3.0
v0.2.0