Storage API Reference

Storage

class neuro_sdk.Storage

Storage subsystem, available as Client.storage.

The subsystem can be used for listing remote storage, uploading and downloading files etc.

Remote filesystem operations

async-with async-for glob(uri: URL, *, dironly: bool = False) → AsyncContextManager[AsyncIterator[URL]]

Glob the given relative pattern uri in the directory represented by this uri, yielding all matching remote files (of any kind):

folder = yarl.URL("storage:folder/*")
async with client.storage.glob(folder) as uris:
    async for uri in uris:
        print(uri)

The “**” pattern means “this directory and all sub-directories, recursively”. In other words, it enables recursive globbing:

folder = yarl.URL("storage:folder/**/*.py")
async with client.storage.glob(folder) as uris:
    async for uri in uris:
        print(uri)

Parameters:

• uri (URL) – a pattern to glob.

• fironly (bool) – search in directory names only, False by default.

Returns:

asynchronous iterator that can be used for enumerating found files and directories.

async-with async-for list(uri: URL) → AsyncContextManager[AsyncIterator[FileStatus]]

List a directory uri on the storage, e.g.:

folder = yarl.URL("storage:folder")
async with client.storage.list(folder) as statuses:
    async for status in statuses:
        print(status.name, status.size)

Parameters:

uri (URL) – directory to list

Returns:

asynchronous iterator which emits FileStatus objects for the directory content.

coroutine mkdir(uri: URL, *, parents: bool = False, exist_ok: bool = False) → None

Create a directory uri. Also create parent directories if parents is True, fail if directory exists and not exist_ok.

Parameters:

• uri (URL) – a path to directory for creation, e.g. yarl.URL("storage:folder/subfolder").

• parents (bool) – create parent directories if they are not exists, raise FileNotFound otherwise (False by default).

• exist_ok (bool) – finish successfully if directory uri already exists, raise FileExistsError otherwise (False by default).

Raises:

FileExistsError if requested directory already exists and exist_ok flag is not set.

Raises:

FileNotFound if parent directories don’t exist and parents flag is not set.

coroutine mv(src: URL, dst: URL) → None

Rename remote file or directory src to dst.

Parameters:

• src (URL) – remote source path, e.g. yarl.URL("storage:folder/file.bin").

• dst (URL) – remote destination path, e.g. yarl.URL("storage:folder/new.bin").

coroutine rm(uri: URL, *, recursive: bool = False) → None

Remove remote file or directory uri.

Parameters:

• uri (URL) – remote path to delete, e.g. yarl.URL("storage:folder/file.bin").

• recursive (bool) – remove a directory recursively with all nested files and folders if True (False by default).

Raises:

IsADirectoryError if uri points on a directory and recursive flag is not set.

coroutine stat(uri: URL) → FileStatus

Return information about uri.

Parameters:

uri (URL) – storage path for requested info, e.g. yarl.URL("storage:folder/file.bin").

Returns:

data structure for given uri, FileStatus object.

coroutine disk_usage(cluster_name: Optional[str] = None, org_name: Optional[str] = None) → DiskUsageInfo

Return information about disk usage in given cluster and organization.

Parameters:

• cluster_name (str) – cluster name to retrieve info. If None current cluster will be used.

• org_name (str) – Organization name to retrieve info. If None current organization will be used.

Returns:

data structure for given cluster and organization, DiskUsageInfo object.

File operations

coroutine create(uri: URL, data: AsyncIterator[bytes]) → None

Create a file on storage under uri name, file it with a content from data asynchronous iterator, e.g.:

async def gen():
    for i in range(10):
        yield str(i) * 10

file = yarl.URL("storage:folder/file.bin")
source = gen()
await client.storage.create(file, source)
await source.aclose()

Parameters:

• uri (URL) – path to created file, e.g. yarl.URL("storage:folder/file.txt").

• data (AsyncIterator[bytes]) – asynchronous iterator used as data provider for file content.

coroutine write(uri: URL, data: bytes, offset: int) → None

Overwrite the part of existing file on storage under uri name.

Parameters:

• uri (URL) – storage path of remote file, e.g. yarl.URL("storage:folder/file.txt").

• data (bytes) – data to be written. Must be non-empty.

• offset (int) – position in file from which to write.

async-with async-for open(uri: URL, offset: int = 0, size: Optional[int] = None) → AsyncContextManager[AsyncIterator[bytes]]

Get the content of remote file uri or its fragment as asynchronous iterator, e.g.:

file = yarl.URL("storage:folder/file.txt")
async with client.storage.open(file) as content:
    async for data in content:
        print(data)

Parameters:

• uri (URL) – storage path of remote file, e.g. yarl.URL("storage:folder/file.txt").

• offset (int) – Position in file from which to read.

• size (int) – Maximal size of the read data. If None read to the end of the file.

Returns:

asynchronous iterator used for retrieving the file content.

Copy operations

coroutine download_dir(src: URL, dst: URL, *, update: bool = False, continue_: bool = False, filter: Optional[Callable[[str], Awaitable[bool]]] = None, progress: Optional[AbstractRecursiveFileProgress] = None) → None:

Recursively download remote directory src to local path dst.

Parameters:

• src (URL) – path on remote storage to download a directory from e.g. yarl.URL("storage:folder").

• dst (URL) – local path to save downloaded directory, e.g. yarl.URL("file:///home/andrew/folder").

• update (bool) – if true, download only when the source file is newer than the destination file or when the destination file is missing.

• continue (bool) – if true, download only the part of the source file past the end of the destination file and append it to the destination file if the destination file is newer and not longer than the source file. Otherwise download and overwrite the whole file.

• filter (Callable[[str], Awaitable[bool]]) – a callback function for determining which files and subdirectories be downloaded. It is called with a relative path of file or directory and if the result is false the file or directory will be skipped.

• progress (AbstractRecursiveFileProgress) – a callback interface for reporting downloading progress, None for no progress report (default).

coroutine download_file(src: URL, dst: URL, *, update: bool = False, continue_: bool = False, progress: Optional[AbstractFileProgress] = None) → None:

Download remote file src to local path dst.

Parameters:

• src (URL) – path on remote storage to download a file from e.g. yarl.URL("storage:folder/file.bin").

• dst (URL) – local path to save downloaded file, e.g. yarl.URL("file:///home/andrew/folder/file.bin").

• update (bool) – if true, download only when the source file is newer than the destination file or when the destination file is missing.

• continue (bool) – if true, download only the part of the source file past the end of the destination file and append it to the destination file if the destination file is newer and not longer than the source file. Otherwise download and overwrite the whole file.

• progress (AbstractFileProgress) – a callback interface for reporting downloading progress, None for no progress report (default).

coroutine upload_dir(src: URL, dst: URL, *, update: bool = False, continue_: bool = False, filter: Optional[Callable[[str], Awaitable[bool]]] = None, ignore_file_names: AbstractSet[str] = frozenset(), progress: Optional[AbstractRecursiveFileProgress] = None) → None:

Recursively upload local directory src to storage URL dst.

Parameters:

• src (URL) – path to uploaded directory on local disk, e.g. yarl.URL("file:///home/andrew/folder").

• dst (URL) – path on remote storage for saving uploading directory e.g. yarl.URL("storage:folder").

• update (bool) – if true, download only when the source file is newer than the destination file or when the destination file is missing.

• continue (bool) – if true, upload only the part of the source file past the end of the destination file and append it to the destination file if the destination file is newer and not longer than the source file. Otherwise upload and overwrite the whole file.

• filter (Callable[[str], Awaitable[bool]]) – a callback function for determining which files and subdirectories be uploaded. It is called with a relative path of file or directory and if the result is false the file or directory will be skipped.

• ignore_file_names (AbstractSet[str]) – a set of names of files which specify filters for skipping files and subdirectories. The format of ignore files is the same as .gitignore.

• progress (AbstractRecursiveFileProgress) – a callback interface for reporting uploading progress, None for no progress report (default).

coroutine upload_file(src: URL, dst: URL, *, update: bool = False, continue_: bool = False, progress: Optional[AbstractFileProgress] = None) → None:

Upload local file src to storage URL dst.

Parameters:

• src (URL) – path to uploaded file on local disk, e.g. yarl.URL("file:///home/andrew/folder/file.txt").

• dst (URL) – path on remote storage for saving uploading file e.g. yarl.URL("storage:folder/file.txt").

• update (bool) – if true, download only when the source file is newer than the destination file or when the destination file is missing.

• continue (bool) – if true, upload only the part of the source file past the end of the destination file and append it to the destination file if the destination file is newer and not longer than the source file. Otherwise upload and overwrite the whole file.

• progress (AbstractFileProgress) – a callback interface for reporting uploading progress, None for no progress report (default).

FileStatus

class neuro_sdk.FileStatus

Read-only dataclass for describing remote entry (file or directory).

modification_time

Modification time in seconds since the epoch, like the value returned from time.time().

name

File or directory name, the last part of path.

permission

Permission (read, write or manage), Action.

path

Path to the entry, str.

size

File size in bytes, int.

type

Entry type, FileStatusType instance.

is_file() → bool

True if type is FileStatusType.FILE

is_dir() → bool

True if type is FileStatusType.DIRECTORY

AbstractFileProgress

class neuro_sdk.AbstractFileProgress

Base class for file upload/download progress, inherited from abc.ABC. User should inherit from this class and override abstract methods to get progress info from Storage.upload_file() and Storage.download_file() calls.

start(data: StorageProgressStart) → None

Called when transmission of the file starts.

Parameters:

data (StorageProgressStart) – data for this event

step(data: StorageProgressStep) → None

Called when transmission of the file progresses (more bytes are transmitted).

Parameters:

data (StorageProgressStep) – data for this event

complete(data: StorageProgressComplete) → None

Called when transmission of the file completes.

Parameters:

data (StorageProgressComplete) – data for this event

AbstractRecursiveFileProgress

class neuro_sdk.AbstractRecursiveFileProgress

Base class for recursive file upload/download progress, inherited from AbstractFileProgress. User should inherit from this class and override abstract methods to get progress info from Storage.upload_dir() and Storage.download_dir() calls.

enter(data: StorageProgressEnterDir) → None

Called when recursive process enters directory.

Parameters:

data (StorageProgressComplete) – data for this event

leave(data: StorageProgressLeaveDir) → None

Called when recursive process leaves directory. All files in that directory are now transmitted.

Parameters:

data (StorageProgressLeaveDir) – data for this event

fail(data: StorageProgressFail) → None

Called when recursive process fails. It can happen because of touching special file (like block device file) or some other reason. Check data.message to get error details.

Parameters:

data (StorageProgressFail) – data for this event

AbstractDeleteProgress

class neuro_sdk.AbstractDeleteProgress

Base class for file/directory delete progress, inherited from abc.ABC. User should inherit from this class and override abstract methods to get progress info from Storage.rm() calls.

delete(data: StorageProgressDelete) → None

Called when single item (either file or directory) was deleted. Directory removal happens after removal of all files and subdirectories is contains.

Parameters:

data (StorageProgressDelete) – data for this event

StorageProgress event classes

class neuro_sdk.StorageProgressStart

src

yarl.URL of source file, e.g. URL("file:/home/user/upload.txt").

dst

yarl.URL of destination file, e.g. URL("storage://cluster/user/upload_to.txt").

size

Size of the transmitted file, in bytes, int.

class neuro_sdk.StorageProgressStep

src

yarl.URL of source file, e.g. URL("file:/home/user/upload.txt").

dst

yarl.URL of destination file, e.g. URL("storage://cluster/user/upload_to.txt").

current

Count of the transmitted bytes, int.

size

Size of the transmitted file, in bytes, int.

class neuro_sdk.StorageProgressComplete

src

yarl.URL of source file, e.g. URL("file:/home/user/upload.txt").

dst

yarl.URL of destination file, e.g. URL("storage://cluster/user/upload_to.txt").

size

Size of the transmitted file, in bytes, int.

class neuro_sdk.StorageProgressEnterDir

src

yarl.URL of source directory, e.g. URL("file:/home/user/upload/").

dst

yarl.URL of destination directory, e.g. URL("storage://cluster/user/upload_to/").

class neuro_sdk.StorageProgressLeaveDir

src

yarl.URL of source directory, e.g. URL("file:/home/user/upload/").

dst

yarl.URL of destination directory, e.g. URL("storage://cluster/user/upload_to/").

class neuro_sdk.StorageProgressFail

src

yarl.URL of source file that triggered error, e.g. URL("file:/dev/sda").

dst

yarl.URL of destination file, e.g. URL("storage://cluster/user/dev/sda.bin").

message

Explanation message for the error, str.

class neuro_sdk.StorageProgressDelete

uri

yarl.URL of the deleted file, e.g. URL("storage://cluster/user/delete_me.txt").

is_dir

True if removed item was a directory; False otherwise. bool

DiskUsageInfo

class neuro_sdk.DiskUsageInfo

Read-only dataclass for describing disk usage in particular cluster

cluster_name

Name of cluster, str.

total

Total storage size in bytes, int.

used

Used storage size in bytes, int.

free

Free storage size in bytes, int.