Changelog ========= 1.5.2 (2023/05) --------------- Deprecations: * Python 3.6 and 3.7 support are removed. 1.5.1 (2021/11) --------------- Improvements: * To improve exceptions readability, airfs now hides its internal exception traceback when converting exception to ``OSError``. Full traceback can be enabled by setting the ``AIRFS_FULLTRACEBACK`` environment variable. Fixes: * Fix ``os.path.realpath`` missing Python 3.10 `strict` argument for local paths. Deprecations: * Warn about Python 3.6 deprecation in next version. 1.5.0 (2020/11) --------------- New storage support: * GitHub (Read only). * The S3 storage now support any supported S3 compatible storage and not only AWS. New standard library equivalent functions: * ``os.readlink``, ``os.symlink``, ``os.path.lexists``, ``os.path.realpath``. New features: * ``airfs.shareable_url`` function to get a shareable URL of an existing object. * airfs now supports a user configuration that can be used to define storage to mount with their parameters. The configuration can be managed using the ``airfs.config`` module. Improvements: * ``airfs.stat`` and ``airfs.lstat`` now returns proper GID and UID, with default to current user values for storage that does not support it. * ``airfs.stat`` and ``airfs.lstat`` ``st_mode`` now include permission bits with default value to ``644`` for storage that does not support it. * ``airfs.stat`` and ``airfs.lstat`` now returns extra storage specific values with their original names instead of ``st_`` prefixed names. To allow use as stat result attributes and ensure consistency, returned names are lower-cased, ``-`` are replaced with ``_`` and other characters are limited to alphanumerical characters. * Add ``airfs.os``, ``airfs.os.path`` and ``airfs.shutil`` namespaces that mimic standards library namespaces. Names that are not available in Airfs fall back to their standard library equivalent. * Add ``st_atime_ns``, ``st_mtime_ns`` and ``st_ctime_ns``, in the ``os.stat_result`` like named tuple returned by ``airfs.stat`` and ``airfs.lstat``. * airfs functions now properly supports the ``follow_symlinks`` argument when storage support symlinks. * ``airfs.listdir`` now properly follows symlinks when storage support symlinks. * airfs can now lazily auto-import a storage from an URL using any root and not only a specific scheme. Only public storage that does not require user specified parameters can be lazily auto-mounted. airfs still try to use the `http` storage to mount unsupported storage like public internet links. * airfs now provides the public base exception ``airfs.AirfsException`` and the public base warning ``airfs.AirfsWarning``. Users can catch them to handle airfs exceptions. * airfs now raises ``airfs.MountException`` on mounting errors instead of ``ValueError``. * airfs now raises ``airfs.ConfigurationException`` on configuration errors. * Refactor ``airfs.io.SystemBase.list_objects`` to improve to support more input file listing and make the API simpler. Remove ``airfs.io.FileSystemBase`` which is now a duplicate. * ``airfs.copy`` and ``airfs.copyfile`` does not raise ``FileNotFoundError`` if the parent directory do not exists and the storage allows to write on a non exiting directory (For instance on blob storages). * airfs exception conversion to OSError is now more strict and convert only excepted exceptions. * Add `Flake8 `_ and `Bandit `_ to the Pytest sequence. * Use `Black `_ for code formatting. Fixes: * airfs now let correctly ``ImportError`` from dependencies be raised instead of hiding it by raising an exception saying that the storage is not available. * ``airfs.io.ObjectBufferedIOBase`` ``read`` and ``readinto`` methods now properly fallback to ``read1`` and ``readinto1`` if the object is not seekable. * airfs now open S3 presigned URLs as standard HTTP URLs instead of trying to use S3 API. * Fix the missing ``follow_symlinks`` argument in ``airfs.copy``. * Fix ``follow_symlinks=False`` argument not raising exception on storage that does not support symlinks. * Fix the missing ``closefd`` and ``opener`` arguments in ``airfs.open``. * Fix the missing keyword-only argument mark in various functions. * airfs now properly raise ``NotImplementedError`` when using ``dir_fd`` on non local paths. * Fix the use of functions that should support file descriptors. Deprecations: * Python 3.5 support is removed. 1.4.0 (2020/01) --------------- Fork from "pycosio" unmaintained project: * Namespace to import changed from `pycosio` to `airfs`. It is possible to upgrade by replacing all ``pycosio`` code occurrences by ``airfs`` or doing ``import airfs as pycosio``. Improvements: * Use `__slots__` on all classes to improve performance and memory usage. Deprecations: * Python 2.7 and 3.4 support are removed. 1.3.3 (2019/07) --------------- Fixes: * Fix package including ``tests/storage_package`` subdirectory due to packaging toolchain issue. 1.3.2 (2019/05) --------------- Fixes: * Fix ``io.BufferedIOBase`` partially read when reading exactly by parts of the buffer size. * Fix bad permission error handling ``airfs.copy``. * Fix bad exceptions risen if no parent directory found using ``airfs.copy``. Contributors: * Thanks to Stewart Adam (stewartadam) for the tests and fixes for Azure. 1.3.1 (2019/04) --------------- Fixes: * Trying to open a file in ``a`` or ``x`` mode now raise ``PermissionError`` if not enough permission to determinate if the file already exists. * Fix ``OSError`` exception conversion in ``copy`` and ``copyfile``. * ``copy`` and ``copyfile`` now tries to copy if no read access but write access, instead of raising ``PermissionError`` because unable to check the parent directory first. * Azure: Fix error when trying to read an existing blog due to bad name handling. * Azure: Fix query string in blob object path because not removed from URL. * Azure: Fix error when opening a blob in ``w`` mode if no permission to read it. Contributors: * Thanks to Stewart Adam (stewartadam) for the tests and fixes for Azure. 1.3.0 (2019/03) --------------- Add support for following storage: * Microsoft Azure Blob Storage * Microsoft Azure File Storage Improvements: * ``io.RawIOBase`` can now be used for storage that supports random write access. * OSS: Copy objects between OSS buckets without copying data on the client when possible. Deprecations: * Warn about Python 3.4 deprecation in next version. Fixes: * Fix unsupported operation not risen in all cases with raw and buffered IO. * Fix call of ``flush()`` in buffered IO. * Fix file methods not translate storage exception into ``OSError``. * Fix file not create on open in write mode (Was only created on flush). * Fix file closed twice when using context manager. * Fix root URL detection in some cases. * Fix too many returned result when listing objects with a count limit. * Fix error when trying to append on a not existing file. * Fix ``io.RawIOBase`` not generating padding when seeking after the end of the file. * OSS: Fix error when listing objects in a not existing directory. * OSS: Fix read error if try to read after the end of the file. * OSS: Fix buffered write minimum buffer size. * OSS: Clean up multipart upload parts on failed uploads. * OSS: Fix error when opening an existing file in 'a' mode. * S3: Fix error when creating a bucket due to an unspecified region. * S3: Fix unprocessed error in listing bucket content of a not existing bucket. * S3: Clean up multipart upload parts on failed uploads. * S3: Fix missing transfer acceleration endpoints. * Swift: Fix error when opening an existing file in 'a' mode. Contributors: * Thanks to Stewart Adam (stewartadam) for the early tests and fixes for Azure. 1.2.0 (2018/10) --------------- New standard library equivalent functions: * ``os.listdir``, ``os.lstat``, ``os.remove``, ``os.rmdir``, ``os.scandir``, ``os.stat``, ``os.unlink``, ``os.path.getctime``, ``os.path.islink``, ``shutil.copyfile``. Improvements: * Copy of objects from and to the same storage is performed directly on remote server if possible. * Now raises ``io.UnsupportedOperation`` if an operation is not compatible with the current storage, this applies to all newly created function and following existing functions: ``getsize``, ``getmtime``, ``mkdir``. Fixes: * ``io.BufferedIOBase.read`` now returns empty bytes instead of raising exception when trying to read if seek already at end of the file. * ``copy`` destination can now be a storage directory and not only a local directory. * ``copy`` now checks if destination parent directory exists and if files are not the same file and raise proper exceptions. * ``mkdir``: missing ``dir_fd`` argument. * ``isdir`` now correctly handle "virtual" directories (Directory that don't exist as a proper object, but exists in another object path). 1.1.0 (2018/10) --------------- New standard library equivalent functions: * ``os.path.exists``, ``os.path.isabs``, ``os.path.isdir``, ``os.path.ismount``, ``os.path.samefile``, ``os.path.splitdrive``, ``os.makedirs``, ``os.mkdir``. Backward incompatible change: * ``mount`` argument ``extra_url_prefix`` is renamed to more relevant and clearer ``extra_root``. Improvements: * No buffer copy when using ``io.BufferedIOBase.read`` with exactly buffer size. This may lead to performance improvement. * Minimum packages versions are set in setup based on packages changelog or date. Fixes: * ``isfile`` now correctly returns ``False`` when used on a directory. * ``relpath`` now keeps ending ``/`` on storage path (Directory marker). 1.0.0 (2018/08) --------------- The first version that implements the core machinery. Provides storage equivalent functions of: * ``open`` / ``io.open``, ``shutil.copy``, ``os.path.getmtime``, ``os.path.getsize``, ``os.path.isfile``, ``os.path.relpath``. Provide storage objects abstract classes with the following interfaces: * ``io.RawIOBase``, ``io.BufferedIOBase``. Add support for following storage: * Alibaba Cloud OSS * AWS S3 * OpenStack Swift Add read-only generic HTTP/HTTPS objects support. Known issues ------------ * Append mode doesn't work with ``ObjectBufferedIOBase``. * Following functions does not follow symlinks yet: * For path target and parents directories: ``airfs.rmdir``, ``airfs.samefile``, ``airfs.copy``, ``airfs.copypath``. * For parent directories: ``airfs.makedirs``, ``airfs.mkdir``, ``airfs.remove``, ``airfs.lstat``, ``airfs.lexists``, ``airfs.islink``