utils

friendly_session(retries=3, total_timeout=300) async

Create an aiohttp session with retry capabilities.

Examples:

Use as async context:

>>> async with friendly_session(retries=5, total_timeout=60) as session:
>>>     r = await session.get("https://example.com/api/data")
>>>     print(r)
<ClientResponse(https://example.com/api/data) [404 Not Found]>
<CIMultiDictProxy('Accept-Ranges': 'bytes', ...

Parameters:

Name	Type	Description	Default
retries	int	The number of retry attempts for failed requests.	3
total_timeout	int	The total timeout for a request in seconds.	300

retrieve_files(urls, save_dir, max_parallel_downloads=5, retries=3, total_timeout=300, desc='Downloading files') async

Retrieve files from a list of URLs and save them to a directory.

Parameters:

Name	Type	Description	Default
urls	Iterable[tuple[str, str]]	A list of tuples, where each tuple contains a URL and a filename.	required
save_dir	Path	The directory to save the downloaded files to.	required
max_parallel_downloads	int	The maximum number of files to download in parallel.	5
retries	int	The number of times to retry a failed download.	3
total_timeout	int	The total timeout for a download in seconds.	300
desc	str	Description for the progress bar.	'Downloading files'

Returns:

Type	Description
list[Path]	A list of paths to the downloaded files.