Skip to main content
Version: v1.1

Inter-Node File Distribution

Introduction​

In a multi-Node Unit, service images are downloaded from the cloud by the Communication Manager (CM) on the Main Node. Secondary Nodes do not have direct cloud connectivity — they receive images and files through the Message Proxy (MP), which bridges the Main Node and Secondary Nodes over the inter-node transport.

This page describes the file distribution pipeline: how CM makes images available via its file server, how MP downloads and unpacks them, and how the filechunker module splits large files into chunks for reliable transport to the Main Node's CM.

Distribution Architecture​

File distribution in AosCore follows a pull-based model. When a Secondary Node's Service Manager (SM) needs a service image, the request flows through MP back to CM, which provides the image via HTTP. MP then downloads, unpacks, and chunks the content for delivery.

The key components involved are:

ComponentLocationRole
CM File ServerMain NodeServes image files over HTTP to other Nodes
CM Image ManagerMain NodeManages image storage and translates file paths to HTTP URLs
MP DownloaderSecondary NodeDownloads files from CM's file server via HTTP
MP Image UnpackerSecondary NodeExtracts and validates OCI service image archives
MP File ChunkerSecondary NodeSplits unpacked image files into chunks for transport

CM File Server​

The Communication Manager runs an embedded HTTP file server that provides access to downloaded service images. This server is implemented using the Poco HTTP library and listens on a configurable port (default: 8080).

How It Works​

  1. CM's Image Manager downloads service images from the cloud and stores them locally
  2. The file server maps local file paths to HTTP URLs using TranslateFilePathURL
  3. When SM on a Secondary Node requests an image, CM provides the HTTP URL pointing to its file server
  4. MP on the Secondary Node uses this URL to download the image

URL Translation​

The file server translates absolute file paths into HTTP URLs relative to its root directory. For example:

Local path:  /var/aos/download/blobs/sha256/abc123...
Server URL:  http://192.168.1.10:8080/blobs/sha256/abc123...

This allows Secondary Nodes to fetch any file that CM has stored, using standard HTTP GET requests.

Image Content Request Flow​

The distribution process is triggered when CM sends an ImageContentRequest message to a Secondary Node's SM (via MP). The flow proceeds as follows:

  1. CM sends request — CM's SM Controller sends an ImageContentRequest to the Secondary Node, containing:

    • url — the HTTP URL where the image can be downloaded (from CM's file server)
    • request_id — a unique identifier for tracking the request
    • content_type — the type of content (e.g., "service" for service images)

  2. MP receives request — The Message Proxy's CM connection intercepts the ImageContentRequest on the secure channel

  3. MP downloads the file — MP uses its downloader to fetch the image archive from the provided URL

  4. MP unpacks the image — The image unpacker extracts and validates the OCI image archive

  5. MP chunks the content — The filechunker splits the unpacked files into fixed-size chunks

  6. MP sends content info — MP sends an ImageContentInfo message back to CM with file metadata (paths, SHA-256 checksums, sizes)

  7. MP sends chunks — MP sends individual ImageContent messages containing the file data in sequential parts

File Chunker​

The filechunker module (aos::mp::filechunker) splits files into fixed-size chunks for reliable transport over the inter-node communication channel. This is necessary because the inter-node transport (vchan or socket) has message size constraints.

Chunking Process​

Given a directory of unpacked image files, the filechunker:

  1. Recursively iterates all files in the directory
  2. For each file, computes a SHA-256 hash for integrity verification
  3. Splits the file content into 1 KB chunks
  4. Produces a ContentInfo structure containing:
    • Image files — metadata for each file (relative path, SHA-256 hash, size)
    • Image contents — the actual data chunks with sequencing information

Chunk Structure​

Each chunk (ImageContent) contains:

FieldDescription
mRequestIDCorrelates chunks to the original request
mRelativePathFile path relative to the image root directory
mPartsCountTotal number of chunks for this file
mPartSequential chunk number (1-based)
mDataRaw file data for this chunk (up to 1024 bytes)

The receiving side reassembles the file by collecting all parts in order and concatenating the data.

Design Rationale​

The 1 KB chunk size is chosen to fit within the inter-node transport's message size limits while maintaining reasonable throughput. The SHA-256 hash per file enables end-to-end integrity verification after reassembly.

Image Unpacker​

The image unpacker module (aos::mp::imageunpacker) receives OCI service image archives and prepares them for local use by the Service Manager.

Unpacking Process​

  1. Extract archive — The tar archive is extracted into a temporary directory within the configured image store
  2. Parse manifest — The manifest.json file is parsed to identify the image config, service config (optional aosService descriptor), and filesystem layers
  3. Validate digests — All content-addressable blobs are verified against their declared SHA-256 digests
  4. Prepare filesystem — The service rootfs layer is extracted from its tar archive and the directory hash is recomputed
  5. Update manifest — The manifest is updated with the new rootfs digest after extraction

OCI Image Structure​

The unpacker expects images in the following OCI-compatible layout:

image/
├── manifest.json          # Image manifest with config and layer references
└── blobs/
    └── sha256/
        ├── <config-hash>  # Image configuration JSON
        ├── <service-hash> # Aos service configuration (optional)
        └── <layer-hash>   # Service rootfs (tar archive or directory)

Content Types​

The unpacker currently supports the "service" content type, which processes standard AosEdge service images. The content type is specified in the original ImageContentRequest from CM.

End-to-End Distribution Sequence​

The complete file distribution sequence for deploying a service to a Secondary Node:

Main Node (CM)                    Secondary Node (MP → SM)
─────────────────                 ────────────────────────
1. Download image from cloud
2. Store in local image store
3. File server makes image
   available via HTTP URL
4. Send ImageContentRequest ──────► 5. Receive request on secure channel
   (url, request_id, type)          6. Download archive from CM file server
                                    7. Unpack OCI image archive
                                    8. Validate image digests
                                    9. Chunk unpacked files (1 KB chunks)
10. Receive ImageContentInfo ◄────── 10. Send file metadata
    (paths, hashes, sizes)              (ImageContentInfo)
11. Receive ImageContent     ◄────── 11. Send data chunks sequentially
    chunks                              (ImageContent messages)
12. Forward to SM for
    service deployment

Security Considerations​

  • TLS-encrypted transport — All inter-node communication channels are secured with mutual TLS using IAM-issued certificates
  • Integrity verification — SHA-256 hashes are computed for every file and verified after transfer
  • Digest validation — OCI image manifests use content-addressable storage; any tampering is detected during the unpack phase
  • Authenticated download — The HTTP connection from MP to CM's file server operates over the secure inter-node channel