Fetcher
Provides an interface for network IO abstraction.
- class FetcherInterface
Defines an interface for abstract network download.
By providing a concrete implementation of the abstract interface, users of the framework can plug-in their preferred/customized network stack.
- download_bytes(url, max_length)
Download bytes from given
url
.Returns the downloaded bytes, otherwise like
download_file()
.- Parameters
url (str) – URL string that represents the location of the file.
max_length (int) – Upper bound of data size in bytes.
- Raises
exceptions.DownloadError – An error occurred during download.
exceptions.DownloadLengthMismatchError – Downloaded bytes exceed
max_length
.exceptions.DownloadHTTPError – An HTTP error code was received.
- Returns
Content of the file in bytes.
- Return type
bytes
- download_file(url, max_length)
Download file from given
url
.It is recommended to use
download_file()
within awith
block to guarantee that allocated file resources will always be released even if download fails.- Parameters
url (str) – URL string that represents the location of the file.
max_length (int) – Upper bound of file size in bytes.
- Raises
exceptions.DownloadError – An error occurred during download.
exceptions.DownloadLengthMismatchError – Downloaded bytes exceed
max_length
.exceptions.DownloadHTTPError – An HTTP error code was received.
- Yields
TemporaryFile
object that points to the contents ofurl
.- Return type
Iterator[IO]
- fetch(url)
Fetches the contents of HTTP/HTTPS
url
from a remote server.- Parameters
url (str) – URL string that represents a file location.
- Raises
exceptions.DownloadError – An error occurred during download.
exceptions.DownloadHTTPError – An HTTP error code was received.
- Returns
Bytes iterator
- Return type
Iterator[bytes]