LR Schedulers

The learning rate scheduler can be changed by adding a SCHEDULER section to the config. The default learning rate scheduler is CosineAnnealing.

Cosine Annealing

SCHEDULER:
    NAME: "CosineAnnealing"
    T_MAX: 10

Cosine Annealing Warm Restarts

SCHEDULER:
    NAME: "CosineAnnealingWithRestarts"
    T_MAX: 10

Exponential Decay

SCHEDULER:
    NAME: "ExponentialDecay"
    DECAY_RATE: 0.96

Identity

SCHEDULER:
  NAME: "IdentityScheduler"

Step

SCHEDULER:
  NAME: "StepScheduler"
  STEP_SIZE: 10
  GAMMA: 0.1