FileClient¶
- class mmengine.fileio.FileClient(backend=None, prefix=None, **kwargs)[source]¶
A general file client to access files in different backends.
The client loads a file or text in a specified backend from its path and returns it as a binary or text file. There are two ways to choose a backend, the name of backend and the prefix of path. Although both of them can be used to choose a storage backend,
backend
has a higher priority that is if they are all set, the storage backend will be chosen by the backend argument. If they are all None, the disk backend will be chosen. Note that It can also register other backend accessor with a given name, prefixes, and backend class. In addition, We use the singleton pattern to avoid repeated object creation. If the arguments are the same, the same object will be returned.Warning
FileClient will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
- Parameters
Examples
>>> # only set backend >>> file_client = FileClient(backend='petrel') >>> # only set prefix >>> file_client = FileClient(prefix='s3') >>> # set both backend and prefix but use backend to choose client >>> file_client = FileClient(backend='petrel', prefix='s3') >>> # if the arguments are the same, the same object is returned >>> file_client1 = FileClient(backend='petrel') >>> file_client1 is file_client True
- client¶
The backend object.
- Type
- get(filepath)[source]¶
Read data from a given
filepath
with ‘rb’ mode.Note
There are two types of return values for
get
, one isbytes
and the other ismemoryview
. The advantage of using memoryview is that you can avoid copying, and if you want to convert it tobytes
, you can use.tobytes()
.- Parameters
filepath (str or Path) – Path to read data.
- Returns
Expected bytes object or a memory view of the bytes object.
- Return type
bytes | memoryview
- get_local_path(filepath)[source]¶
Download data from
filepath
and write the data to local path.get_local_path
is decorated bycontxtlib.contextmanager()
. It can be called withwith
statement, and when exists from thewith
statement, the temporary path will be released.Note
If the
filepath
is a local path, just return itself.Warning
get_local_path
is an experimental interface that may change in the future.- Parameters
filepath (str or Path) – Path to be read data.
- Return type
Generator[Union[str, pathlib.Path], None, None]
Examples
>>> file_client = FileClient(prefix='s3') >>> with file_client.get_local_path('s3://bucket/abc.jpg') as path: ... # do something here
- Yields
Iterable[str] – Only yield one path.
- Parameters
filepath (Union[str, pathlib.Path]) –
- Return type
Generator[Union[str, pathlib.Path], None, None]
- classmethod infer_client(file_client_args=None, uri=None)[source]¶
Infer a suitable file client based on the URI and arguments.
- Parameters
file_client_args (dict, optional) – Arguments to instantiate a FileClient. Default: None.
uri (str | Path, optional) – Uri to be parsed that contains the file prefix. Default: None.
- Return type
Examples
>>> uri = 's3://path/of/your/file' >>> file_client = FileClient.infer_client(uri=uri) >>> file_client_args = {'backend': 'petrel'} >>> file_client = FileClient.infer_client(file_client_args)
- Returns
Instantiated FileClient object.
- Return type
- Parameters
file_client_args (Optional[dict]) –
uri (Optional[Union[str, pathlib.Path]]) –
- join_path(filepath, *filepaths)[source]¶
Concatenate all file paths.
Join one or more filepath components intelligently. The return value is the concatenation of filepath and any members of *filepaths.
- Parameters
filepath (str or Path) – Path to be concatenated.
filepaths (Union[str, pathlib.Path]) –
- Returns
The result of concatenation.
- Return type
- list_dir_or_file(dir_path, list_dir=True, list_file=True, suffix=None, recursive=False)[source]¶
Scan a directory to find the interested directories or files in arbitrary order.
Note
list_dir_or_file()
returns the path relative todir_path
.- Parameters
dir_path (str | Path) – Path of the directory.
list_dir (bool) – List the directories. Default: True.
list_file (bool) – List the path of files. Default: True.
suffix (str or tuple[str], optional) – File suffix that we are interested in. Default: None.
recursive (bool) – If set to True, recursively scan the directory. Default: False.
- Yields
Iterable[str] – A relative path to
dir_path
.- Return type
Iterator[str]
- static parse_uri_prefix(uri)[source]¶
Parse the prefix of a uri.
- Parameters
uri (str | Path) – Uri to be parsed that contains the file prefix.
- Return type
Optional[str]
Examples
>>> FileClient.parse_uri_prefix('s3://path/of/your/file') 's3'
- Returns
Return the prefix of uri if the uri contains ‘://’ else
None
.- Return type
str | None
- Parameters
uri (Union[str, pathlib.Path]) –
- put(obj, filepath)[source]¶
Write data to a given
filepath
with ‘wb’ mode.Note
put
should create a directory if the directory offilepath
does not exist.
- put_text(obj, filepath)[source]¶
Write data to a given
filepath
with ‘w’ mode.Note
put_text
should create a directory if the directory offilepath
does not exist.
- classmethod register_backend(name, backend=None, force=False, prefixes=None)[source]¶
Register a backend to FileClient.
This method can be used as a normal class method or a decorator.
class NewBackend(BaseStorageBackend): def get(self, filepath): return filepath def get_text(self, filepath): return filepath FileClient.register_backend('new', NewBackend)
or
@FileClient.register_backend('new') class NewBackend(BaseStorageBackend): def get(self, filepath): return filepath def get_text(self, filepath): return filepath
- Parameters
name (str) – The name of the registered backend.
backend (class, optional) – The backend class to be registered, which must be a subclass of
BaseStorageBackend
. When this method is used as a decorator, backend is None. Defaults to None.force (bool, optional) – Whether to override the backend if the name has already been registered. Defaults to False.
prefixes (str or list[str] or tuple[str], optional) – The prefixes of the registered storage backend. Default: None. New in version 1.3.15.