airfs.storage.s3
Amazon Web Service S3 (and compatible) storage.
New in version 1.0.0.
Mount
AWS S3 can be mounted using the airfs mount
function.
storage_parameters
contain a sub-directory session
that await arguments to pass
to boto3.session.Session(**session_parameters)
from the boto3
Python library.
It can also include a sub-directory client
that is used to pass arguments to
boto3.session.Session.client('s3', **client_parameters)
.
This example shows the mount of S3 with the minimal configuration:
import airfs
# Mount S3 manually (Minimal configuration)
airfs.mount(
storage='s3',
storage_parameters=dict(
# "boto3.session.Session" arguments
session=dict(
aws_access_key_id='my_access_key',
aws_secret_access_key='my_secret_key',
region_name='my_region_name'
)
)
)
# Call of airfs on a S3 object.
with airfs.open('s3://my_bucket/my_object', 'rt') as file:
text = file.read()
Automatic mount
It is not required to mount S3 explicitly when using airfs on a host configured to handle AWS S3 access (Through IAM policy, configuration files, environment variables, …).
In this case, mounting is done transparently on the first call of a airfs function on an S3 object and no configuration or extra steps are required:
import airfs
# Call of airfs on a S3 object: Mounting is done transparently.
with airfs.open('s3://my_bucket/my_bucket', 'rt') as file:
text = file.read()
S3 compatible storage mount
It is possible to use any S3 compatible storage with airfs. This kind of storage needs to be mounted with the endpoint_url storage parameter specified. This endpoint must also be use as path root (instead of the classical s3://).
import airfs
# Mount S3 compatible storage
airfs.mount(
storage='s3',
storage_parameters=dict(
# "boto3.client" arguments
client=dict(
aws_access_key_id='my_access_key',
aws_secret_access_key='my_secret_key',
region_name='my_region_name',
endpoint_url="https://s3.my_storage.com/"
)
)
)
# Call of airfs on a S3 compatible object.
with airfs.open('https://s3.my_storage.com/my_bucket/my_object', 'rt') as file:
text = file.read()
Limitation
Only one S3 configuration can be mounted simultaneously for AWS, and one for each other specified endpoint_url.
Files objects classes
Amazon Web Services S3.
- class airfs.storage.s3.S3BufferedIO(*args, **kwargs)[source]
Buffered binary S3 Object I/O.
- MINIMUM_BUFFER_SIZE = 5242880
Minimal buffer_size in bytes (S3 multipart upload minimal part size)
- close()
Flush the write buffers of the stream if applicable and close the object.
- detach()
Disconnect this buffer from its underlying raw stream and return it.
After the raw stream has been detached, the buffer is in an unusable state.
- fileno()
Returns underlying file descriptor if one exists.
OSError is raised if the IO object does not use a file descriptor.
- flush()
Flush the write buffers of the stream if applicable.
- isatty()
Return whether this is an ‘interactive’ stream.
Return False if it can’t be determined.
- property mode
The mode.
- Returns:
Mode.
- Return type:
str
- property name
The file name.
- Returns:
Name.
- Return type:
str
- peek(size=-1)
Return bytes from the stream without advancing the position.
- Parameters:
size (int) – Number of bytes to read. -1 to read the full stream.
- Returns:
bytes read
- Return type:
bytes
- property raw
The underlying raw stream.
- Returns:
Raw stream.
- Return type:
ObjectRawIOBase subclass
- read(size=-1)
Read the object content.
Read and return up to size bytes, with at most one call to the underlying raw stream.
Use at most one call to the underlying raw stream’s read method.
- Parameters:
size (int) – Number of bytes to read. -1 to read the stream until the end.
- Returns:
Object content
- Return type:
bytes
- read1(size=-1)
Read the object content.
Read and return up to size bytes, with at most one call to the underlying raw stream.
Use at most one call to the underlying raw stream’s read method.
- Parameters:
size (int) – Number of bytes to read. -1 to read the stream until the end.
- Returns:
Object content
- Return type:
bytes
- readable()
Return True if the stream can be read from.
If False, read() will raise OSError.
- Returns:
Supports reading.
- Return type:
bool
- readinto(b)
Read the object content into a buffer.
Read bytes into a pre-allocated, writable bytes-like object b, and return the number of bytes read.
- Parameters:
b (bytes-like object) – buffer.
- Returns:
number of bytes read
- Return type:
int
- readinto1(b)
Read the object content into a buffer.
Read bytes into a pre-allocated, writable bytes-like object b, and return the number of bytes read.
Use at most one call to the underlying raw stream’s readinto method.
- Parameters:
b (bytes-like object) – buffer.
- Returns:
number of bytes read
- Return type:
int
- readline(size=-1, /)
Read and return a line from the stream.
If size is specified, at most size bytes will be read.
The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.
- readlines(hint=-1, /)
Return a list of lines from the stream.
hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.
- seek(offset, whence=0)
Change the stream position to the given byte offset.
- Parameters:
offset – Offset is interpreted relative to the position indicated by whence.
whence – The default value for whence is SEEK_SET. Values are: SEEK_SET or 0 – Start of the stream (the default); offset should be zero or positive SEEK_CUR or 1 – Current stream position; offset may be negative SEEK_END or 2 – End of the stream; offset is usually negative
- Returns:
The new absolute position.
- Return type:
int
- seekable()
Return True if the stream supports random access.
If False, seek(), tell() and truncate() will raise OSError.
- Returns:
Supports random access.
- Return type:
bool
- tell()
Return the current stream position.
- Returns:
Stream position.
- Return type:
int
- truncate()
Truncate file to size bytes.
File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.
- writable()
Return True if the stream supports writing.
If False, write() and truncate() will raise OSError.
- Returns:
Supports writing.
- Return type:
bool
- write(b)
Write into the object.
Write the given bytes-like object, b, to the underlying raw stream, and return the number of bytes written.
- Parameters:
b (bytes-like object) – Bytes to write.
- Returns:
The number of bytes written.
- Return type:
int
- writelines(lines, /)
Write a list of lines to stream.
Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.
- class airfs.storage.s3.S3RawIO(name, mode='r', storage_parameters=None, **kwargs)[source]
Binary S3 Object I/O.
- Parameters:
name (path-like object) – URL or path to the file which will be opened.
mode (str) – The mode can be ‘r’, ‘w’, ‘a’ for reading (default), writing or appending.
storage_parameters (dict) – Boto3 Session keyword arguments. This is generally AWS credentials and configuration. This dict should contain two sub-dicts: ‘session’: That pass its arguments to “boto3.session.Session”; ‘client’: That pass its arguments to “boto3.session.Session.client”. May be optional if already configured on host.
unsecure (bool) – If True, disables TLS/SSL to improve transfer performance. But makes connection unsecure.
- close()
Flush the write buffers of the stream if applicable and close the object.
- fileno()
Returns underlying file descriptor if one exists.
OSError is raised if the IO object does not use a file descriptor.
- flush()
Flush.
Flush the write buffers of the stream if applicable and save the object on the storage.
- isatty()
Return whether this is an ‘interactive’ stream.
Return False if it can’t be determined.
- property mode
The mode.
- Returns:
Mode.
- Return type:
str
- property name
The file name.
- Returns:
Name.
- Return type:
str
- readable()
Return True if the stream can be read from.
If False, read() will raise OSError.
- Returns:
Supports reading.
- Return type:
bool
- readall()
Read and return all the bytes from the stream until EOF.
- Returns:
Object content
- Return type:
bytes
- readinto(b)
Read the object content into a buffer.
Read bytes into a pre-allocated, writable bytes-like object b, and return the number of bytes read.
- Parameters:
b (bytes-like object) – buffer.
- Returns:
number of bytes read
- Return type:
int
- readline(size=-1, /)
Read and return a line from the stream.
If size is specified, at most size bytes will be read.
The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.
- readlines(hint=-1, /)
Return a list of lines from the stream.
hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.
- seek(offset, whence=0)
Change the stream position to the given byte offset.
- Parameters:
offset (int) – Offset is interpreted relative to the position indicated by whence.
whence (int) – The default value for whence is SEEK_SET. Values are: SEEK_SET or 0 – Start of the stream (the default); offset should be zero or positive SEEK_CUR or 1 – Current stream position; offset may be negative SEEK_END or 2 – End of the stream; offset is usually negative
- Returns:
The new absolute position.
- Return type:
int
- seekable()
Return True if the stream supports random access.
If False, seek(), tell() and truncate() will raise OSError.
- Returns:
Supports random access.
- Return type:
bool
- tell()
Return the current stream position.
- Returns:
Stream position.
- Return type:
int
- truncate()
Truncate file to size bytes.
File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.
- writable()
Return True if the stream supports writing.
If False, write() and truncate() will raise OSError.
- Returns:
Supports writing.
- Return type:
bool
- write(b)
Write into the object.
Write the given bytes-like object, b, to the underlying raw stream, and return the number of bytes written.
- Parameters:
b (bytes-like object) – Bytes to write.
- Returns:
The number of bytes written.
- Return type:
int
- writelines(lines, /)
Write a list of lines to stream.
Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.