FileClient¶

class mmengine.fileio.FileClient(backend=None, prefix=None, **kwargs)[source]¶

A general file client to access files in different backends.

The client loads a file or text in a specified backend from its path and returns it as a binary or text file. There are two ways to choose a backend, the name of backend and the prefix of path. Although both of them can be used to choose a storage backend, backend has a higher priority that is if they are all set, the storage backend will be chosen by the backend argument. If they are all None, the disk backend will be chosen. Note that It can also register other backend accessor with a given name, prefixes, and backend class. In addition, We use the singleton pattern to avoid repeated object creation. If the arguments are the same, the same object will be returned.

Warning

FileClient will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io

Parameters

backend (str, optional) – The storage backend type. Options are “disk”, “memcached”, “lmdb”, “http” and “petrel”. Default: None.
prefix (str, optional) – The prefix of the registered storage backend. Options are “s3”, “http”, “https”. Default: None.

Examples

>>> # only set backend
>>> file_client = FileClient(backend='petrel')
>>> # only set prefix
>>> file_client = FileClient(prefix='s3')
>>> # set both backend and prefix but use backend to choose client
>>> file_client = FileClient(backend='petrel', prefix='s3')
>>> # if the arguments are the same, the same object is returned
>>> file_client1 = FileClient(backend='petrel')
>>> file_client1 is file_client
True

client¶

The backend object.

Type: BaseStorageBackend

exists(filepath)[source]¶

Check whether a file path exists.

Parameters: filepath (str or Path) – Path to be checked whether exists.
Returns: Return True if filepath exists, False otherwise.
Return type: bool

get(filepath)[source]¶

Read data from a given filepath with ‘rb’ mode.

Note

There are two types of return values for get, one is bytes and the other is memoryview. The advantage of using memoryview is that you can avoid copying, and if you want to convert it to bytes, you can use .tobytes().

Parameters: filepath (str or Path) – Path to read data.
Returns: Expected bytes object or a memory view of the bytes object.
Return type: bytes | memoryview

get_local_path(filepath)[source]¶

Download data from filepath and write the data to local path.

get_local_path is decorated by contxtlib.contextmanager(). It can be called with with statement, and when exists from the with statement, the temporary path will be released.

Note

If the filepath is a local path, just return itself.

Warning

get_local_path is an experimental interface that may change in the future.

Parameters: filepath (str or Path) – Path to be read data.
Return type: Generator[Union[str, pathlib.Path], None, None]

Examples

>>> file_client = FileClient(prefix='s3')
>>> with file_client.get_local_path('s3://bucket/abc.jpg') as path:
...     # do something here

Yields: Iterable[str] – Only yield one path.
Parameters: filepath (Union[str, pathlib.Path]) –
Return type: Generator[Union[str, pathlib.Path], None, None]

get_text(filepath, encoding='utf-8')[source]¶

Read data from a given filepath with ‘r’ mode.

Parameters

filepath (str or Path) – Path to read data.
encoding (str) – The encoding format used to open the filepath. Default: ‘utf-8’.

Returns

Expected text reading from filepath.

Return type

str

classmethod infer_client(file_client_args=None, uri=None)[source]¶

Infer a suitable file client based on the URI and arguments.

Parameters

file_client_args (dict, optional) – Arguments to instantiate a FileClient. Default: None.
uri (str | Path, optional) – Uri to be parsed that contains the file prefix. Default: None.

Return type

mmengine.fileio.file_client.FileClient

Examples

>>> uri = 's3://path/of/your/file'
>>> file_client = FileClient.infer_client(uri=uri)
>>> file_client_args = {'backend': 'petrel'}
>>> file_client = FileClient.infer_client(file_client_args)

Returns

Instantiated FileClient object.

Return type

FileClient

Parameters

file_client_args (Optional[dict]) –
uri (Optional[Union[str, pathlib.Path]]) –

isdir(filepath)[source]¶

Check whether a file path is a directory.

Parameters: filepath (str or Path) – Path to be checked whether it is a directory.
Returns: Return True if filepath points to a directory, False otherwise.
Return type: bool

isfile(filepath)[source]¶

Check whether a file path is a file.

Parameters: filepath (str or Path) – Path to be checked whether it is a file.
Returns: Return True if filepath points to a file, False otherwise.
Return type: bool

join_path(filepath, *filepaths)[source]¶

Concatenate all file paths.

Join one or more filepath components intelligently. The return value is the concatenation of filepath and any members of *filepaths.

Parameters

filepath (str or Path) – Path to be concatenated.
filepaths (Union[str, pathlib.Path]) –

Returns

The result of concatenation.

Return type

str

list_dir_or_file(dir_path, list_dir=True, list_file=True, suffix=None, recursive=False)[source]¶

Scan a directory to find the interested directories or files in arbitrary order.

Note

list_dir_or_file() returns the path relative to dir_path.

Parameters

dir_path (str | Path) – Path of the directory.
list_dir (bool) – List the directories. Default: True.
list_file (bool) – List the path of files. Default: True.
suffix (str or tuple[str], optional) – File suffix that we are interested in. Default: None.
recursive (bool) – If set to True, recursively scan the directory. Default: False.

Yields

Iterable[str] – A relative path to dir_path.

Return type

Iterator[str]

static parse_uri_prefix(uri)[source]¶

Parse the prefix of a uri.

Parameters: uri (str | Path) – Uri to be parsed that contains the file prefix.
Return type: Optional[str]

Examples

>>> FileClient.parse_uri_prefix('s3://path/of/your/file')
's3'

Returns: Return the prefix of uri if the uri contains ‘://’ else None.
Return type: str | None
Parameters: uri (Union[str, pathlib.Path]) –

put(obj, filepath)[source]¶

Write data to a given filepath with ‘wb’ mode.

Note

put should create a directory if the directory of filepath does not exist.

Parameters

obj (bytes) – Data to be written.
filepath (str or Path) – Path to write data.

Return type

None

put_text(obj, filepath)[source]¶

Write data to a given filepath with ‘w’ mode.

Note

put_text should create a directory if the directory of filepath does not exist.

Parameters

obj (str) – Data to be written.
filepath (str or Path) – Path to write data.
encoding (str, optional) – The encoding format used to open the filepath. Default: ‘utf-8’.

Return type

None

classmethod register_backend(name, backend=None, force=False, prefixes=None)[source]¶

Register a backend to FileClient.

This method can be used as a normal class method or a decorator.

class NewBackend(BaseStorageBackend):

    def get(self, filepath):
        return filepath

    def get_text(self, filepath):
        return filepath

FileClient.register_backend('new', NewBackend)

or

@FileClient.register_backend('new')
class NewBackend(BaseStorageBackend):

    def get(self, filepath):
        return filepath

    def get_text(self, filepath):
        return filepath

Parameters

name (str) – The name of the registered backend.
backend (class, optional) – The backend class to be registered, which must be a subclass of BaseStorageBackend. When this method is used as a decorator, backend is None. Defaults to None.
force (bool, optional) – Whether to override the backend if the name has already been registered. Defaults to False.
prefixes (str or list[str] or tuple[str], optional) – The prefixes of the registered storage backend. Default: None. New in version 1.3.15.

remove(filepath)[source]¶

Remove a file.

Parameters: filepath (str, Path) – Path to be removed.
Return type: None