Visualization¶
1 Overall Design¶
Visualization provides an intuitive explanation of the training and testing process of the deep learning model. In OpenMMLab, we expect the visualization module to meet the following requirements:
Provides rich out-of-the-box features that can meet most computer vision visualization tasks.
Versatile, expandable, and can be customized easily
Able to visualize at anywhere in the training and testing process.
Unified APIs for all OpenMMLab libraries, which is convenient for users to understand and use.
Based on the above requirements, we proposed the Visualizer
and various VisBackend
such as LocalVisBackend
, WandbVisBackend
, and TensorboardVisBackend
in OpenMMLab 2.0. The visualizer could not only visualize the image data, but also things like configurations, scalars, and model structure.
For convenience, the APIs provided by the
Visualizer
implement the drawing and storage functions. As an internal property ofVisualizer
,VisBackend
will be called byVisualizer
to write data to different backends.Considering that you may want to write data to multiple backends after drawing,
Visualizer
can be configured with multiple backends. When the user calls the storage API of theVisualizer
, it will traverse and call all the specified APIs ofVisBackend
internally.
The UML diagram of the two is as follows.
2 Visualizer¶
The external interface of Visualizer
can be divided into three categories.
Drawing APIs
draw_bboxes draws a single or multiple bounding boxes
draw_points draws a single or multiple points
draw_texts draws a single or multiple text boxes
draw_lines draws a single or multiple line segments
draw_circles draws a single or multiple circles
draw_polygons draws a single or multiple polygons
draw_binary_masks draws single or multiple binary masks
draw_featmap draws feature map (static method)
The above APIs can be called in a chain except for draw_featmap
because the image size may change after this method is called. To avoid confusion, draw_featmap
is a static method.
Storage APIs
add_config writes configuration to a specific storage backend
add_graph writes model graph to a specific storage backend
add_image writes image to a specific storage backend
add_scalar writes scalar to a specific storage backend
add_scalars writes multiple scalars to a specific storage backend at once
add_datasample the abstract interface for each repositories to draw data sample
Interfaces beginning with the add
prefix represent storage APIs. [datasample] (./data_element.md
)is the unified interface of each downstream repository in the OpenMMLab 2.0, and add_datasample
can process the data sample directly .
Other APIs
set_image sets the original image data, the default input image format is RGB
get_image gets the image data in Numpy format after drawing, the default output format is RGB
show for visualization
get_backend gets a specific storage backend by name
close closes all resources, including
VisBackend
For more details, you can refer to Visualizer Tutorial.
3 VisBackend¶
After drawing, the drawn data can be stored in multiple visualization storage backends. To unify the interfaces, MMEngine provides an abstract class, BaseVisBackend
, and some commonly used backends such as LocalVisBackend
, WandbVisBackend
, and TensorboardVisBackend
.
The main interfaces and properties of BaseVisBackend
are as follows:
add_config writes configuration to a specific storage backend
add_graph writes model graph to a specific backend
add_image writes image to a specific backend
add_scalar writes scalar to a specific backend
add_scalars writes multiple scalars to a specific backend at once
close closes the resource that has been opened
experiment writes backend objects, such as WandB objects and Tensorboard objects
BaseVisBackend
defines five common data writing interfaces. Some writing backends are very powerful, such as WandB, which could write tables and videos. Users can directly obtain the experiment
object for such needs and then call native APIs of the corresponding backend. LocalVisBackend
, WandbVisBackend
, and TensorboardVisBackend
are all inherited from BaseVisBackend
and implement corresponding storage functions according to their features. Users can also customize BaseVisBackend
to extend the storage backends and implement custom storage requirements.
For more details, you can refer to Storage Backend Tutorial.