airfs.io

Cloud storage abstract IO classes.

These abstract classes are used as base to implement storage-specific IO classes.

class airfs.io.ObjectBufferedIOBase(name, mode='r', buffer_size=None, max_buffers=0, max_workers=None, **kwargs)

Base class for buffered binary storage object I/O.

close()[source]: Flush the write buffers of the stream if applicable and close the object.

detach()

Disconnect this buffer from its underlying raw stream and return it.

After the raw stream has been detached, the buffer is in an unusable state.

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

flush()[source]: Flush the write buffers of the stream if applicable.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

property mode

The mode.

Returns:: Mode.
Return type:: str

property name

The file name.

Returns:: Name.
Return type:: str

peek(size=-1)[source]

Return bytes from the stream without advancing the position.

Parameters:: size (int) – Number of bytes to read. -1 to read the full stream.
Returns:: bytes read
Return type:: bytes

property raw

The underlying raw stream.

Returns:: Raw stream.
Return type:: ObjectRawIOBase subclass

read(size=-1)[source]

Read the object content.

Read and return up to size bytes, with at most one call to the underlying raw stream.

Use at most one call to the underlying raw stream’s read method.

Parameters:: size (int) – Number of bytes to read. -1 to read the stream until the end.
Returns:: Object content
Return type:: bytes

read1(size=-1)[source]

Read the object content.

Read and return up to size bytes, with at most one call to the underlying raw stream.

Use at most one call to the underlying raw stream’s read method.

Parameters:: size (int) – Number of bytes to read. -1 to read the stream until the end.
Returns:: Object content
Return type:: bytes

readable()

Return True if the stream can be read from.

If False, read() will raise OSError.

Returns:: Supports reading.
Return type:: bool

readinto(b)[source]

Read the object content into a buffer.

Read bytes into a pre-allocated, writable bytes-like object b, and return the number of bytes read.

Parameters:: b (bytes-like object) – buffer.
Returns:: number of bytes read
Return type:: int

readinto1(b)[source]

Read the object content into a buffer.

Read bytes into a pre-allocated, writable bytes-like object b, and return the number of bytes read.

Use at most one call to the underlying raw stream’s readinto method.

Parameters:: b (bytes-like object) – buffer.
Returns:: number of bytes read
Return type:: int

readline(size=-1, /)

Read and return a line from the stream.

If size is specified, at most size bytes will be read.

The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.

readlines(hint=-1, /)

Return a list of lines from the stream.

hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

seek(offset, whence=0)[source]

Change the stream position to the given byte offset.

Parameters:

offset – Offset is interpreted relative to the position indicated by whence.
whence – The default value for whence is SEEK_SET. Values are: SEEK_SET or 0 – Start of the stream (the default); offset should be zero or positive SEEK_CUR or 1 – Current stream position; offset may be negative SEEK_END or 2 – End of the stream; offset is usually negative

Returns:

The new absolute position.

Return type:

int

seekable()

Return True if the stream supports random access.

If False, seek(), tell() and truncate() will raise OSError.

Returns:: Supports random access.
Return type:: bool

tell()

Return the current stream position.

Returns:: Stream position.
Return type:: int

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return True if the stream supports writing.

If False, write() and truncate() will raise OSError.

Returns:: Supports writing.
Return type:: bool

write(b)[source]

Write into the object.

Write the given bytes-like object, b, to the underlying raw stream, and return the number of bytes written.

Parameters:: b (bytes-like object) – Bytes to write.
Returns:: The number of bytes written.
Return type:: int

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

class airfs.io.ObjectBufferedIORandomWriteBase(name, mode='r', buffer_size=None, max_buffers=0, max_workers=None, **kwargs)

Base class for buffered binary storage object I/O with partial flush support.

close(): Flush the write buffers of the stream if applicable and close the object.

detach()

Disconnect this buffer from its underlying raw stream and return it.

After the raw stream has been detached, the buffer is in an unusable state.

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

flush(): Flush the write buffers of the stream if applicable.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

property mode

The mode.

Returns:: Mode.
Return type:: str

property name

The file name.

Returns:: Name.
Return type:: str

peek(size=-1)

Return bytes from the stream without advancing the position.

Parameters:: size (int) – Number of bytes to read. -1 to read the full stream.
Returns:: bytes read
Return type:: bytes

property raw

The underlying raw stream.

Returns:: Raw stream.
Return type:: ObjectRawIOBase subclass

read(size=-1)

Read the object content.

Read and return up to size bytes, with at most one call to the underlying raw stream.

Use at most one call to the underlying raw stream’s read method.

Parameters:: size (int) – Number of bytes to read. -1 to read the stream until the end.
Returns:: Object content
Return type:: bytes

read1(size=-1)

Read the object content.

Read and return up to size bytes, with at most one call to the underlying raw stream.

Use at most one call to the underlying raw stream’s read method.

Parameters:: size (int) – Number of bytes to read. -1 to read the stream until the end.
Returns:: Object content
Return type:: bytes

readable()

Return True if the stream can be read from.

If False, read() will raise OSError.

Returns:: Supports reading.
Return type:: bool

readinto(b)

Read the object content into a buffer.

Read bytes into a pre-allocated, writable bytes-like object b, and return the number of bytes read.

Parameters:: b (bytes-like object) – buffer.
Returns:: number of bytes read
Return type:: int

readinto1(b)

Read the object content into a buffer.

Read bytes into a pre-allocated, writable bytes-like object b, and return the number of bytes read.

Use at most one call to the underlying raw stream’s readinto method.

Parameters:: b (bytes-like object) – buffer.
Returns:: number of bytes read
Return type:: int

readline(size=-1, /)

Read and return a line from the stream.

If size is specified, at most size bytes will be read.

The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.

readlines(hint=-1, /)

Return a list of lines from the stream.

hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

seek(offset, whence=0)

Change the stream position to the given byte offset.

Parameters:

offset – Offset is interpreted relative to the position indicated by whence.
whence – The default value for whence is SEEK_SET. Values are: SEEK_SET or 0 – Start of the stream (the default); offset should be zero or positive SEEK_CUR or 1 – Current stream position; offset may be negative SEEK_END or 2 – End of the stream; offset is usually negative

Returns:

The new absolute position.

Return type:

int

seekable()

Return True if the stream supports random access.

If False, seek(), tell() and truncate() will raise OSError.

Returns:: Supports random access.
Return type:: bool

tell()

Return the current stream position.

Returns:: Stream position.
Return type:: int

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return True if the stream supports writing.

If False, write() and truncate() will raise OSError.

Returns:: Supports writing.
Return type:: bool

write(b)

Write into the object.

Write the given bytes-like object, b, to the underlying raw stream, and return the number of bytes written.

Parameters:: b (bytes-like object) – Bytes to write.
Returns:: The number of bytes written.
Return type:: int

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

class airfs.io.ObjectRawIOBase(name, mode='r', storage_parameters=None, **kwargs)

Base class for binary storage object I/O.

In write mode, this class needs enough memory to store the entire object to write. In append mode, the storage object is read and stored in memory on instantiation. For big objects use ObjectBufferedIOBase that can performs operations with less memory.

In read mode, this class random access to the storage object and requires only the accessed data size in memory.

close()[source]: Flush the write buffers of the stream if applicable and close the object.

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

flush()[source]

Flush.

Flush the write buffers of the stream if applicable and save the object on the storage.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

property mode

The mode.

Returns:: Mode.
Return type:: str

property name

The file name.

Returns:: Name.
Return type:: str

readable()

Return True if the stream can be read from.

If False, read() will raise OSError.

Returns:: Supports reading.
Return type:: bool

readall()[source]

Read and return all the bytes from the stream until EOF.

Returns:: Object content
Return type:: bytes

readinto(b)[source]

Read the object content into a buffer.

Read bytes into a pre-allocated, writable bytes-like object b, and return the number of bytes read.

Parameters:: b (bytes-like object) – buffer.
Returns:: number of bytes read
Return type:: int

readline(size=-1, /)

Read and return a line from the stream.

If size is specified, at most size bytes will be read.

The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.

readlines(hint=-1, /)

Return a list of lines from the stream.

hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

seek(offset, whence=0)[source]

Change the stream position to the given byte offset.

Parameters:

offset (int) – Offset is interpreted relative to the position indicated by whence.
whence (int) – The default value for whence is SEEK_SET. Values are: SEEK_SET or 0 – Start of the stream (the default); offset should be zero or positive SEEK_CUR or 1 – Current stream position; offset may be negative SEEK_END or 2 – End of the stream; offset is usually negative

Returns:

The new absolute position.

Return type:

int

seekable()

Return True if the stream supports random access.

If False, seek(), tell() and truncate() will raise OSError.

Returns:: Supports random access.
Return type:: bool

tell()

Return the current stream position.

Returns:: Stream position.
Return type:: int

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return True if the stream supports writing.

If False, write() and truncate() will raise OSError.

Returns:: Supports writing.
Return type:: bool

write(b)[source]

Write into the object.

Write the given bytes-like object, b, to the underlying raw stream, and return the number of bytes written.

Parameters:: b (bytes-like object) – Bytes to write.
Returns:: The number of bytes written.
Return type:: int

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

class airfs.io.ObjectRawIORandomWriteBase(name, mode='r', storage_parameters=None, **kwargs)

Base class for raw binary storage object I/O with partial flush support.

Objects that support flushing parts of file instead of requiring flushing the full file at once.

close(): Flush the write buffers of the stream if applicable and close the object.

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

flush()[source]

Flush.

Flush the write buffers of the stream if applicable and save the object on the storage.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

property mode

The mode.

Returns:: Mode.
Return type:: str

property name

The file name.

Returns:: Name.
Return type:: str

readable()

Return True if the stream can be read from.

If False, read() will raise OSError.

Returns:: Supports reading.
Return type:: bool

readall()

Read and return all the bytes from the stream until EOF.

Returns:: Object content
Return type:: bytes

readinto(b)

Read the object content into a buffer.

Read bytes into a pre-allocated, writable bytes-like object b, and return the number of bytes read.

Parameters:: b (bytes-like object) – buffer.
Returns:: number of bytes read
Return type:: int

readline(size=-1, /)

Read and return a line from the stream.

If size is specified, at most size bytes will be read.

The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.

readlines(hint=-1, /)

Return a list of lines from the stream.

hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

seek(offset, whence=0)[source]

Change the stream position to the given byte offset.

Parameters:

offset (int) – Offset is interpreted relative to the position indicated by whence.
whence (int) – The default value for whence is SEEK_SET. Values are: SEEK_SET or 0 – Start of the stream (the default); offset should be zero or positive SEEK_CUR or 1 – Current stream position; offset may be negative SEEK_END or 2 – End of the stream; offset is usually negative

Returns:

The new absolute position.

Return type:

int

seekable()

Return True if the stream supports random access.

If False, seek(), tell() and truncate() will raise OSError.

Returns:: Supports random access.
Return type:: bool

tell()

Return the current stream position.

Returns:: Stream position.
Return type:: int

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return True if the stream supports writing.

If False, write() and truncate() will raise OSError.

Returns:: Supports writing.
Return type:: bool

write(b)

Write into the object.

Write the given bytes-like object, b, to the underlying raw stream, and return the number of bytes written.

Parameters:: b (bytes-like object) – Bytes to write.
Returns:: The number of bytes written.
Return type:: int

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

class airfs.io.SystemBase(storage_parameters=None, unsecure=False, roots=None, **_)

Cloud storage system handler.

This class subclasses are not intended to be public and are implementation details.

This base system is for Object storage that does not handle files with a true hierarchy like file systems. Directories are virtual with this kind of storage.

property client

Storage client.

Returns:: client

copy(src, dst, other_system=None)[source]

Copy an object of the same storage.

Parameters:

src (str) – Path or URL.
dst (str) – Path or URL.
other_system (airfs._core.io_system.SystemBase subclass) – Another storage system. May be required for some storage.

ensure_dir_path(path, relative=False)[source]

Ensure the path is a dir path.

Should end with ‘/’ except for schemes and locators.

Parameters:

path (str) – Path or URL.
relative (bool) – Path is relative to current root.

Returns:

dir path

Return type:

path

exists(path=None, client_kwargs=None, assume_exists=None, header=None, follow_symlinks=None)[source]

Return True if the path refers to an existing path.

Parameters:

path (str) – Path or URL.
client_kwargs (dict) – Client arguments.
assume_exists (bool or None) – This value defines the value to return in case there is no enough permission to determinate the existing status of the file. If set to None, the permission exception is reraised (Default behavior). if set to True or False, return this value.
header (dict) – Object header.
follow_symlinks (bool) – Follow symlinks.

Returns:

True if exists.

Return type:

bool

abstract get_client_kwargs(path)[source]

Get base keyword arguments for the client for a specific path.

Parameters:: path (str) – Absolute path or URL.
Returns:: client args
Return type:: dict

getctime(path=None, client_kwargs=None, header=None)[source]

Return the creation time of path.

Parameters:

path (str) – File path or URL.
client_kwargs (dict) – Client arguments.
header (dict) – Object header.

Returns:

The number of seconds since the epoch (see the time module).

Return type:

float

getmtime(path=None, client_kwargs=None, header=None)[source]

Return the time of last access of path.

Parameters:

path (str) – File path or URL.
client_kwargs (dict) – Client arguments.
header (dict) – Object header.

Returns:

The number of seconds since the epoch (see the time module).

Return type:

float

getsize(path=None, client_kwargs=None, header=None)[source]

Return the size, in bytes, of path.

Parameters:

path (str) – File path or URL.
client_kwargs (dict) – Client arguments.
header (dict) – Object header.

Returns:

Size in bytes.

Return type:

int

head(path=None, client_kwargs=None, header=None)[source]

Returns object HTTP header.

Parameters:

path (str) – Path or URL.
client_kwargs (dict) – Client arguments.
header (dict) – Object header.

Returns:

HTTP header.

Return type:

dict

is_abs(path)[source]

Return True if the path is absolute in this storage.

Parameters:: path (str) – Path or URL.
Returns:: True if absolute path.
Return type:: bool

is_locator(path, relative=False)[source]

Returns True if the path refer to a locator.

Depending on the storage, locator may be a bucket or container name, a hostname,…

Parameters:

path (str) – path or URL.
relative (bool) – Path is relative to current root.

Returns:

True if locator.

Return type:

bool

isdir(path=None, client_kwargs=None, virtual_dir=True, assume_exists=None, header=None, follow_symlinks=None)[source]

Return True if the path is an existing directory.

Parameters:

path (str) – Path or URL.
client_kwargs (dict) – Client arguments.
virtual_dir (bool) – If True, checks if directory exists virtually if an object path if not exists as a specific object.
assume_exists (bool or None) – This value defines the value to return in case there is no enough permission to determinate the existing status of the file. If set to None, the permission exception is reraised (Default behavior). if set to True or False, return this value.
header (dict) – Object header.
follow_symlinks (bool) – Follow symlinks.

Returns:

True if directory exists.

Return type:

bool

isfile(path=None, client_kwargs=None, assume_exists=None, header=None, follow_symlinks=None)[source]

Return True if the path is an existing regular file.

Parameters:

path (str) – Path or URL.
client_kwargs (dict) – Client arguments.
assume_exists (bool or None) – This value defines the value to return in case there is no enough permission to determinate the existing status of the file. If set to None, the permission exception is reraised (Default behavior). if set to True or False, return this value.
header (dict) – Object header.
follow_symlinks (bool) – Follow symlinks.

Returns:

True if file exists.

Return type:

bool

islink(path=None, client_kwargs=None, header=None)[source]

Returns True if the object is a symbolic link.

Parameters:

path (str) – File path or URL.
client_kwargs (dict) – Client arguments.
header (dict) – Object header.

Returns:

True if the object is Symlink.

Return type:

bool

list_objects(path='', relative=False, first_level=False, max_results=None)[source]

List objects.

Returns object path (relative to input “path”) and object headers.

Parameters:

path (str) – Path or URL.
relative (bool) – Path is relative to current root.
first_level (bool) – If True, returns only first level objects. Else, returns full tree.
max_results (int) – If specified, the maximum result count returned.

Yields:

tuple – object path str, object header dict

make_dir(path, relative=False)[source]

Make a directory.

Parameters:

path (str) – Path or URL.
relative (bool) – Path is relative to current root.

read_link(path=None, client_kwargs=None, header=None)[source]

Return the path linked by the symbolic link.

Parameters:

path (str) – File path or URL.
client_kwargs (dict) – Client arguments.
header (dict) – Object header.

Returns:

Path.

Return type:

str

relpath(path)[source]

Get the path relative to storage.

Parameters:: path (str) – Absolute path or URL.
Returns:: relative path.
Return type:: str

remove(path, relative=False)[source]

Remove an object.

Parameters:

path (str) – Path or URL.
relative (bool) – Path is relative to current root.

resolve(path=None, client_kwargs=None, header=None, follow_symlinks=None)[source]

Follow symlinks and return input arguments updated for target.

Parameters:

path (str) – File path or URL.
client_kwargs (dict) – Client arguments.
header (dict) – Object header.
follow_symlinks (bool) – If True, follow symlink if any. If False, return input directly if symlink is supported by storage, else raise NotImplementedError. If None, the same as False but returns input instead of raising exception.

Returns:

path, client_kwargs, headers of the target.

Return type:

tuple

Raises:

ObjectNotImplementedError – follow_symlink is False on storage that does not support symlink.

property roots

Return URL roots for this storage.

Returns:: URL roots
Return type:: tuple of str

shareable_url(path, expires_in)[source]

Get a shareable URL for the specified path.

Parameters:

path (str) – Path or URL.
expires_in (int) – Expiration in seconds.

Returns:

Shareable URL.

Return type:

str

split_locator(path)[source]

Split the path into a pair (locator, path).

Parameters:: path (str) – Absolute path or URL.
Returns:: locator, path.
Return type:: tuple of str

stat(path=None, client_kwargs=None, header=None, follow_symlinks=None)[source]

Get the status of an object.

Parameters:

path (str) – File path or URL.
client_kwargs (dict) – Client arguments.
header (dict) – Object header.
follow_symlinks (bool) – Follow symlinks.

Returns:

Stat result object. Follow the “os.stat_result” specification: and may contain storage dependent extra entries.

Return type:

namedtuple

property storage

Storage name.

Returns:: Storage
Return type:: str

property storage_parameters

Storage parameters.

Returns:: Storage parameters
Return type:: dict

symlink(target, path=None, client_kwargs=None)[source]

Creates a symbolic link to target.

Parameters:

target (str) – Target path or URL.
path (str) – File path or URL.
client_kwargs (dict) – Client arguments.