Visualize Training Logs

MMEngine integrates experiment management tools such as TensorBoard, Weights & Biases (WandB), MLflow, ClearML, Neptune, DVCLive and Aim, making it easy to track and visualize metrics like loss and accuracy.

Below, we’ll show you how to configure an experiment management tool in just one line, based on the example from 15 minutes to get started with MMEngine.

TensorBoard

Configure the visualizer in the initialization parameters of the Runner, and set vis_backends to TensorboardVisBackend.

runner = Runner(
    model=MMResNet50(),
    work_dir='./work_dir',
    train_dataloader=train_dataloader,
    optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)),
    train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1),
    val_dataloader=val_dataloader,
    val_cfg=dict(),
    val_evaluator=dict(type=Accuracy),
    visualizer=dict(type='Visualizer', vis_backends=[dict(type='TensorboardVisBackend')]),
)
runner.train()

WandB

Before using WandB, you need to install the wandb dependency library and log in to WandB.

pip install wandb
wandb login

Configure the visualizer in the initialization parameters of the Runner, and set vis_backends to WandbVisBackend.

runner = Runner(
    model=MMResNet50(),
    work_dir='./work_dir',
    train_dataloader=train_dataloader,
    optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)),
    train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1),
    val_dataloader=val_dataloader,
    val_cfg=dict(),
    val_evaluator=dict(type=Accuracy),
    visualizer=dict(type='Visualizer', vis_backends=[dict(type='WandbVisBackend')]),
)
runner.train()

You can click on WandbVisBackend API to view the configurable parameters for WandbVisBackend. For example, the init_kwargs parameter will be passed to the wandb.init method.

runner = Runner(
    ...
    visualizer=dict(
        type='Visualizer',
        vis_backends=[
            dict(
                type='WandbVisBackend',
                init_kwargs=dict(project='toy-example')
            ),
        ],
    ),
    ...
)
runner.train()

MLflow (WIP)

ClearML

Before using ClearML, you need to install the clearml dependency library and refer to Connect ClearML SDK to the Server for configuration.

pip install clearml
clearml-init

Configure the visualizer in the initialization parameters of the Runner, and set vis_backends to ClearMLVisBackend.

runner = Runner(
    model=MMResNet50(),
    work_dir='./work_dir',
    train_dataloader=train_dataloader,
    optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)),
    train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1),
    val_dataloader=val_dataloader,
    val_cfg=dict(),
    val_evaluator=dict(type=Accuracy),
    visualizer=dict(type='Visualizer', vis_backends=[dict(type='ClearMLVisBackend')]),
)
runner.train()

Neptune

Before using Neptune, you need to install neptune dependency library and refer to Neptune.AI for configuration.

pip install neptune

Configure the Runner in the initialization parameters of the Runner, and set vis_backends to NeptuneVisBackend.

runner = Runner(
    model=MMResNet50(),
    work_dir='./work_dir',
    train_dataloader=train_dataloader,
    optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)),
    train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1),
    val_dataloader=val_dataloader,
    val_cfg=dict(),
    val_evaluator=dict(type=Accuracy),
    visualizer=dict(type='Visualizer', vis_backends=[dict(type='NeptuneVisBackend')]),
)
runner.train()

Please note: If the project and api_token are not specified, neptune will be set to offline mode and the generated files will be saved to the local .neptune file. It is recommended to specify the project and api_token during initialization as shown below.

runner = Runner(
    ...
    visualizer=dict(
        type='Visualizer',
        vis_backends=[
            dict(
                type='NeptuneVisBackend',
                init_kwargs=dict(project='workspace-name/project-name',
                                 api_token='your api token')
            ),
        ],
    ),
    ...
)
runner.train()

More initialization configuration parameters are available at neptune.init_run API.

DVCLive

Before using DVCLive, you need to install dvclive dependency library and refer to iterative.ai for configuration. Common configurations are as follows:

pip install dvclive
cd ${WORK_DIR}
git init
dvc init
git commit -m "DVC init"

Configure the Runner in the initialization parameters of the Runner, and set vis_backends to DVCLiveVisBackend.

runner = Runner(
    model=MMResNet50(),
    work_dir='./work_dir_dvc',
    train_dataloader=train_dataloader,
    optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)),
    train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1),
    val_dataloader=val_dataloader,
    val_cfg=dict(),
    val_evaluator=dict(type=Accuracy),
    visualizer=dict(type='Visualizer', vis_backends=[dict(type='DVCLiveVisBackend')]),
)
runner.train()

Note

Recommend not to set work_dir as work_dirs. Or DVC will give a warning WARNING:dvclive:Error in cache: bad DVC file name 'work_dirs\xxx.dvc' is git-ignored if you run experiments in a OpenMMLab’s repo.

Open the report.html file under work_dir_dvc, and you will see the visualization as shown in the following image.

You can also configure a VSCode extension of DVC to visualize the training process.

More initialization configuration parameters are available at DVCLive API Reference.

Aim

Before using Aim, you need to install aim dependency library.

pip install aim

Configure the Runner in the initialization parameters of the Runner, and set vis_backends to AimVisBackend.

runner = Runner(
    model=MMResNet50(),
    work_dir='./work_dir',
    train_dataloader=train_dataloader,
    optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)),
    train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1),
    val_dataloader=val_dataloader,
    val_cfg=dict(),
    val_evaluator=dict(type=Accuracy),
    visualizer=dict(type='Visualizer', vis_backends=[dict(type='AimVisBackend')]),
)
runner.train()

In the terminal, use the following command,

aim up

or in the Jupyter Notebook, use the following command,

%load_ext aim
%aim up

to launch the Aim UI as shown below.

Initialization configuration parameters are available at Aim SDK Reference.