Shortcuts

FileClient

class mmengine.fileio.FileClient(backend=None, prefix=None, **kwargs)[source]

A general file client to access files in different backends.

The client loads a file or text in a specified backend from its path and returns it as a binary or text file. There are two ways to choose a backend, the name of backend and the prefix of path. Although both of them can be used to choose a storage backend, backend has a higher priority that is if they are all set, the storage backend will be chosen by the backend argument. If they are all None, the disk backend will be chosen. Note that It can also register other backend accessor with a given name, prefixes, and backend class. In addition, We use the singleton pattern to avoid repeated object creation. If the arguments are the same, the same object will be returned.

Warning

FileClient will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io

Parameters
  • backend (str, optional) – The storage backend type. Options are “disk”, “memcached”, “lmdb”, “http” and “petrel”. Defaults to None.

  • prefix (str, optional) – The prefix of the registered storage backend. Options are “s3”, “http”, “https”. Defaults to None.

Examples

>>> # only set backend
>>> file_client = FileClient(backend='petrel')
>>> # only set prefix
>>> file_client = FileClient(prefix='s3')
>>> # set both backend and prefix but use backend to choose client
>>> file_client = FileClient(backend='petrel', prefix='s3')
>>> # if the arguments are the same, the same object is returned
>>> file_client1 = FileClient(backend='petrel')
>>> file_client1 is file_client
True
client

The backend object.

Type

BaseStorageBackend

exists(filepath)[source]

Check whether a file path exists.

Parameters

filepath (str or Path) – Path to be checked whether exists.

Returns

Return True if filepath exists, False otherwise.

Return type

bool

get(filepath)[source]

Read data from a given filepath with ‘rb’ mode.

Note

There are two types of return values for get, one is bytes and the other is memoryview. The advantage of using memoryview is that you can avoid copying, and if you want to convert it to bytes, you can use .tobytes().

Parameters

filepath (str or Path) – Path to read data.

Returns

Expected bytes object or a memory view of the bytes object.

Return type

bytes | memoryview

get_local_path(filepath)[source]

Download data from filepath and write the data to local path.

get_local_path is decorated by contxtlib.contextmanager(). It can be called with with statement, and when exists from the with statement, the temporary path will be released.

Note

If the filepath is a local path, just return itself.

Warning

get_local_path is an experimental interface that may change in the future.

Parameters

filepath (str or Path) – Path to be read data.

Return type

Generator[Union[str, pathlib.Path], None, None]

Examples

>>> file_client = FileClient(prefix='s3')
>>> with file_client.get_local_path('s3://bucket/abc.jpg') as path:
...     # do something here
Yields

Iterable[str] – Only yield one path.

Parameters

filepath (Union[str, pathlib.Path]) –

Return type

Generator[Union[str, pathlib.Path], None, None]

get_text(filepath, encoding='utf-8')[source]

Read data from a given filepath with ‘r’ mode.

Parameters
  • filepath (str or Path) – Path to read data.

  • encoding (str) – The encoding format used to open the filepath. Defaults to ‘utf-8’.

Returns

Expected text reading from filepath.

Return type

str

classmethod infer_client(file_client_args=None, uri=None)[source]

Infer a suitable file client based on the URI and arguments.

Parameters
  • file_client_args (dict, optional) – Arguments to instantiate a FileClient. Defaults to None.

  • uri (str | Path, optional) – Uri to be parsed that contains the file prefix. Defaults to None.

Return type

mmengine.fileio.file_client.FileClient

Examples

>>> uri = 's3://path/of/your/file'
>>> file_client = FileClient.infer_client(uri=uri)
>>> file_client_args = {'backend': 'petrel'}
>>> file_client = FileClient.infer_client(file_client_args)
Returns

Instantiated FileClient object.

Return type

FileClient

Parameters
isdir(filepath)[source]

Check whether a file path is a directory.

Parameters

filepath (str or Path) – Path to be checked whether it is a directory.

Returns

Return True if filepath points to a directory, False otherwise.

Return type

bool

isfile(filepath)[source]

Check whether a file path is a file.

Parameters

filepath (str or Path) – Path to be checked whether it is a file.

Returns

Return True if filepath points to a file, False otherwise.

Return type

bool

join_path(filepath, *filepaths)[source]

Concatenate all file paths.

Join one or more filepath components intelligently. The return value is the concatenation of filepath and any members of *filepaths.

Parameters
Returns

The result of concatenation.

Return type

str

list_dir_or_file(dir_path, list_dir=True, list_file=True, suffix=None, recursive=False)[source]

Scan a directory to find the interested directories or files in arbitrary order.

Note

list_dir_or_file() returns the path relative to dir_path.

Parameters
  • dir_path (str | Path) – Path of the directory.

  • list_dir (bool) – List the directories. Defaults to True.

  • list_file (bool) – List the path of files. Defaults to True.

  • suffix (str or tuple[str], optional) – File suffix that we are interested in. Defaults to None.

  • recursive (bool) – If set to True, recursively scan the directory. Defaults to False.

Yields

Iterable[str] – A relative path to dir_path.

Return type

Iterator[str]

static parse_uri_prefix(uri)[source]

Parse the prefix of a uri.

Parameters

uri (str | Path) – Uri to be parsed that contains the file prefix.

Return type

Optional[str]

Examples

>>> FileClient.parse_uri_prefix('s3://path/of/your/file')
's3'
Returns

Return the prefix of uri if the uri contains ‘://’ else None.

Return type

str | None

Parameters

uri (Union[str, pathlib.Path]) –

put(obj, filepath)[source]

Write data to a given filepath with ‘wb’ mode.

Note

put should create a directory if the directory of filepath does not exist.

Parameters
  • obj (bytes) – Data to be written.

  • filepath (str or Path) – Path to write data.

Return type

None

put_text(obj, filepath)[source]

Write data to a given filepath with ‘w’ mode.

Note

put_text should create a directory if the directory of filepath does not exist.

Parameters
  • obj (str) – Data to be written.

  • filepath (str or Path) – Path to write data.

  • encoding (str, optional) – The encoding format used to open the filepath. Defaults to ‘utf-8’.

Return type

None

classmethod register_backend(name, backend=None, force=False, prefixes=None)[source]

Register a backend to FileClient.

This method can be used as a normal class method or a decorator.

class NewBackend(BaseStorageBackend):

    def get(self, filepath):
        return filepath

    def get_text(self, filepath):
        return filepath

FileClient.register_backend('new', NewBackend)

or

@FileClient.register_backend('new')
class NewBackend(BaseStorageBackend):

    def get(self, filepath):
        return filepath

    def get_text(self, filepath):
        return filepath
Parameters
  • name (str) – The name of the registered backend.

  • backend (class, optional) – The backend class to be registered, which must be a subclass of BaseStorageBackend. When this method is used as a decorator, backend is None. Defaults to None.

  • force (bool, optional) – Whether to override the backend if the name has already been registered. Defaults to False.

  • prefixes (str or list[str] or tuple[str], optional) – The prefixes of the registered storage backend. Defaults to None. New in version 1.3.15.

remove(filepath)[source]

Remove a file.

Parameters

filepath (str, Path) – Path to be removed.

Return type

None

Read the Docs v: v0.8.3
Versions
latest
stable
v0.8.3
v0.8.2
v0.8.1
v0.8.0
v0.7.4
v0.7.3
v0.7.2
v0.7.1
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.3.0
v0.2.0
Downloads
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.