class mmengine.optim.CosineAnnealingParamScheduler(optimizer, param_name, T_max=None, eta_min=0.0, begin=0, end=1000000000, last_step=- 1, by_epoch=True, verbose=False)[源代码]

Set the parameter value of each parameter group using a cosine annealing schedule, where \(\eta_{max}\) is set to the initial value and \(T_{cur}\) is the number of epochs since the last restart in SGDR:

\[\begin{split}\begin{aligned} \eta_t & = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right), & T_{cur} \neq (2k+1)T_{max}; \\ \eta_{t+1} & = \eta_{t} + \frac{1}{2}(\eta_{max} - \eta_{min}) \left(1 - \cos\left(\frac{1}{T_{max}}\pi\right)\right), & T_{cur} = (2k+1)T_{max}. \end{aligned}\end{split}\]

Notice that because the schedule is defined recursively, the parameter value can be simultaneously modified outside this scheduler by other operators. If the parameter value is set solely by this scheduler, the parameter value at each step becomes:

\[\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right)\]

It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts. Note that this only implements the cosine annealing part of SGDR, and not the restarts.

  • optimizer (Optimizer or OptimWrapper) – optimizer or Wrapped optimizer.

  • param_name (str) – Name of the parameter to be adjusted, such as lr, momentum.

  • T_max (int, optional) – Maximum number of iterations. If not specified, use end - begin. Defaults to None.

  • eta_min (float) – Minimum parameter value. Defaults to 0.

  • begin (int) – Step at which to start updating the parameters. Defaults to 0.

  • end (int) – Step at which to stop updating the parameters. Defaults to INF.

  • last_step (int) – The index of last step. Used for resume without state dict. Defaults to -1.

  • by_epoch (bool) – Whether the scheduled parameters are updated by epochs. Defaults to True.

  • verbose (bool) – Whether to print the value for each update. Defaults to False.

classmethod build_iter_from_epoch(*args, T_max, begin=0, end=1000000000, by_epoch=True, epoch_length=None, **kwargs)[源代码]

Build an iter-based instance of this scheduler from an epoch-based config.

Read the Docs v: v0.2.0
On Read the Docs
Project Home

Free document hosting provided by Read the Docs.