Shortcuts

OneCycleParamScheduler

class mmengine.optim.OneCycleParamScheduler(optimizer, param_name, eta_max=0, total_steps=None, pct_start=0.3, anneal_strategy='cos', div_factor=25.0, final_div_factor=10000.0, three_phase=False, begin=0, end=1000000000, last_step=- 1, by_epoch=True, verbose=False)[source]

Sets the parameters of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates.

The 1cycle learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training.

This scheduler is not chainable.

Note also that the total number of steps in the cycle can be determined in one of two ways (listed in order of precedence):

  1. A value for total_steps is explicitly provided.

  2. If total_steps is not defined, begin and end of the ParamSchedul will works for it. In this case, the number of total steps is inferred by total_steps = end - begin

The default behaviour of this scheduler follows the fastai implementation of 1cycle, which claims that “unpublished work has shown even better results by using only two phases”. To mimic the behaviour of the original paper instead, set three_phase=True.

Parameters
  • optimizer (Optimizer) – Wrapped optimizer.

  • param_name (str) – Name of the parameter to be adjusted, such as lr, momentum.

  • eta_max (float or list) – Upper parameter value boundaries in the cycle for each parameter group.

  • total_steps (int) – The total number of steps in the cycle. Note that if a value is not provided here, then it will be equal to end - begin. Defaults to None

  • pct_start (float) – The percentage of the cycle (in number of steps) spent increasing the learning rate. Defaults to 0.3

  • anneal_strategy (str) – {‘cos’, ‘linear’} Specifies the annealing strategy: “cos” for cosine annealing, “linear” for linear annealing. Defaults to ‘cos’

  • div_factor (float) – Determines the initial learning rate via initial_param = eta_max/div_factor Defaults to 25

  • final_div_factor (float) – Determines the minimum learning rate via eta_min = initial_param/final_div_factor Defaults to 1e4

  • three_phase (bool) – If True, use a third phase of the schedule to annihilate the learning rate according to ‘final_div_factor’ instead of modifying the second phase (the first two phases will be symmetrical about the step indicated by ‘pct_start’).

  • last_step (int) – The index of last step. Used for resume without state dict. Defaults to -1.

  • by_epoch (bool) – Whether the scheduled parameters are updated by epochs. Defaults to True.

  • verbose (bool) – Whether to print the value for each update. Defaults to False.

  • begin (int) –

  • end (int) –

classmethod build_iter_from_epoch(*args, begin=0, end=1000000000, total_steps=None, by_epoch=True, epoch_length=None, **kwargs)[source]

Build an iter-based instance of this scheduler from an epoch-based config.

Read the Docs v: latest
Versions
latest
stable
v0.8.4
v0.8.3
v0.8.2
v0.8.1
v0.8.0
v0.7.4
v0.7.3
v0.7.2
v0.7.1
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.3.0
v0.2.0
Downloads
epub
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.