Downloader

Introduction

The Downloader is a shared infrastructure module that provides reliable file retrieval over HTTP/HTTPS. It is used by the Service Manager's Image Manager to download service images and by the Communication Manager's Image Manager to download Deployment Bundles from cloud-provided blob URLs. The Message Proxy also uses the Downloader for retrieving images in multi-Node configurations.

The module implements production-grade reliability features including automatic retry with exponential backoff, download resume via HTTP range requests, periodic progress reporting through the alerts subsystem, and per-download cancellation. It is built on libcurl for HTTP operations and supports file:// URIs for local file copy.

Architecture

The Downloader follows AosCore's two-layer design pattern:

• Interface (aos_core_lib_cpp/src/core/common/downloader/itf/downloader.hpp) — defines the DownloaderItf abstract interface with Download and Cancel methods
• Implementation (aos_core_cpp/src/common/downloader/downloader.cpp) — provides the concrete Downloader class using libcurl

This separation allows components to depend on the interface for testability while the application wires in the concrete implementation at startup.

Interface Contract

The Downloader interface exposes two operations:

Method	Parameters	Description
Download	digest, URL, output path	Downloads a file from the given URL to the specified local path, identified by its content digest
Cancel	digest	Cancels an in-progress download identified by its digest

Each download is uniquely identified by its digest (the content hash of the expected file). This allows the system to track, cancel, and report progress for individual downloads even when multiple downloads run concurrently.

Download Lifecycle

A download progresses through the following stages:

1. Initiation — the caller provides a digest, source URL, and destination path
2. Duplicate check — if a download with the same digest is already in progress, the request is rejected with an eAlreadyExist error
3. Retry loop — the download is attempted up to the maximum retry count
4. Protocol dispatch — file:// URIs are handled by direct filesystem copy; HTTP/HTTPS URLs use libcurl
5. Resume detection — for HTTP/HTTPS, if a partial file exists at the destination, the server is probed for range-request support
6. Transfer — data is streamed to the output file with progress callbacks
7. Completion — on success, a finished alert is sent; on failure after all retries, the last error is returned

Retry and Backoff

When a download attempt fails, the Downloader retries with exponential backoff:

Parameter	Value	Description
Maximum retries	3	Total number of download attempts before giving up
Initial delay	1 second	Wait time before the first retry
Maximum delay	5 seconds	Upper bound on backoff delay
Backoff factor	2×	Each retry doubles the delay (1s → 2s → 4s, capped at 5s)
Connection timeout	10 seconds	Maximum time to establish a connection

The retry loop respects both shutdown signals and cancellation requests — if either occurs during the backoff wait, the download terminates immediately without waiting for the delay to expire.

Between retries, the Downloader checks whether the partially downloaded file can be resumed. If the server supports HTTP range requests (Accept-Ranges: bytes), the next attempt continues from where the previous one left off rather than restarting from the beginning.

Resume Support

The Downloader implements download resume for interrupted transfers:

1. Before starting a download, it checks if a partial file already exists at the destination path
2. If a partial file exists, it sends an HTTP HEAD request to check for Accept-Ranges: bytes in the response headers
3. If the server supports range requests, the download resumes from the existing file size using CURLOPT_RESUME_FROM_LARGE
4. If the server does not support range requests, the file is overwritten from the beginning

This mechanism is particularly valuable for large service images where network interruptions would otherwise require re-downloading the entire file.

Progress Reporting

The Downloader reports download progress through the AosCore alerts subsystem. Progress alerts are sent at a configurable interval (default: 30 seconds) and include:

Field	Description
Digest	Content digest identifying the download
URL	Source URL being downloaded
State	Current download state (started, paused, interrupted, finished)
Downloaded bytes	Number of bytes received so far (including any resumed portion)
Total bytes	Total expected file size (as reported by the server)
Reason	Human-readable description (populated on interruption)
Error	Error details (populated on failure)

Download States

State	Meaning
started	Download is actively transferring data (sent with periodic progress updates)
paused	Download is temporarily suspended
interrupted	Download failed due to a network error or HTTP error response
finished	Download completed successfully

Progress reporting is optional — the Downloader can be initialized without an alerts sender, in which case no progress alerts are emitted. The Service Manager initializes the Downloader without alerts, while the Communication Manager provides an alerts sender for cloud-visible progress tracking.

Cancellation

Downloads can be cancelled at any time using the Cancel method with the download's digest:

• Each active download has an associated atomic cancel flag
• The libcurl progress callback checks this flag on every progress tick — when set, it returns a non-zero value that causes libcurl to abort the transfer
• The retry loop also checks the cancel flag before each attempt and during backoff waits
• After cancellation, the Download call returns an eRuntime error indicating the download was cancelled
• Cancelling a non-existent digest returns an eNotFound error

This design allows immediate cancellation without waiting for network timeouts or retry delays.

Concurrent Downloads

The Downloader supports multiple simultaneous downloads, each identified by a unique digest:

• A map of cancel flags tracks all active downloads, protected by a mutex
• Each download operates independently with its own retry state and progress context
• Attempting to start a second download with the same digest as an active download returns eAlreadyExist
• Cancellation targets a specific digest without affecting other active downloads

Local File Copy

For file:// URIs, the Downloader bypasses the HTTP stack entirely and performs a direct filesystem copy. This is used in development and testing scenarios where images are available on the local filesystem. The copy operation:

• Extracts the file path from the URI
• Verifies the source file exists
• Copies the file content to the destination using stream operations
• Does not support resume or progress reporting (the copy is atomic)

Integration Points

Service Manager

SM's Image Manager uses the Downloader to retrieve service image blobs (layers, manifests, configs) from URLs provided in the desired state. The SM initializes the Downloader without an alerts sender — progress is not reported to the cloud for SM-initiated downloads.

Communication Manager

CM's Image Manager uses the Downloader to retrieve Deployment Bundle blobs from cloud-provided URLs. CM initializes the Downloader with an alerts sender, enabling download progress to be reported back to the cloud as part of the update status flow.

Message Proxy

In multi-Node configurations, the Message Proxy uses the Downloader to retrieve images from the main Node's file server, enabling secondary Nodes to obtain service images without direct cloud access.

Dependencies

The Downloader implementation depends on:

Dependency	Purpose
libcurl	HTTP/HTTPS transfer operations, range requests, progress callbacks
Poco::Foundation	URI parsing for protocol dispatch (file://, http://, https://)
Alerts subsystem	Optional progress reporting through SenderItf
Filesystem (std)	File existence checks, size queries, file I/O

Related Pages

• Common Infrastructure — overview of all shared modules
• OCI Image Format — image structure that the Downloader retrieves

• Space Allocator — manages disk space for downloaded files

• SM Image Manager — primary consumer of the Downloader for service images
• CM Update Manager — coordinates downloads as part of the update flow
• Image Deployment Pipeline — end-to-end flow from download to launch