Optimizers

The optimizer can be changed by adding a OPTIMIZER section to the config. The default optimizer is Adam.

Adam

OPTIMIZER:
  NAME: "Adam"
  LR: 0.001
  WEIGHT_DECAY: 1.0E-5
  EPS: 1.0E-8

Reference

Adam: A Method for Stochastic Optimization

AdamW

OPTIMIZER:
    NAME: "AdamW"
    LR: 0.001
    WEIGHT_DECAY: 1.0E-5
    EPS: 1.0E-8

Reference

Decoupled Weight Decay Regularization

SAM

OPTIMIZER:
  NAME: "SAM"
  LR: 0.001
  RHO: 0.05
  BASE_OPTIMIZER_NAME: "Adam"
  WEIGHT_DECAY: 1.0E-5
  EPS: 1.0E-8

Reference

Sharpness-Aware Minimization for Efficiently Improving Generalization

SGD

OPTIMIZER:
  NAME: "SGD"
  LR: 0.001
  MOMENTUM: 0.9