Skip to main content
Version: v1.1

Message Proxy

Introduction​

The Message Proxy (MP) is the component that enables multi-Node Units by bridging communication between Secondary Nodes and the Main Node's Communication Manager. It runs exclusively on Secondary Nodes, providing them with indirect access to cloud connectivity and centralized orchestration without requiring each Node to maintain its own cloud connection.

From the Unit's perspective, the cloud communicates with a single entity — the Unit itself. The Message Proxy is an internal implementation detail that routes messages between Nodes within the Unit, invisible to external systems.

This page covers the MP's architecture, its subcomponents, the transport layer it uses, and how it fits into the overall multi-Node communication model.

Role in the Architecture​

In a multi-Node Unit, only the Main Node runs the Communication Manager (CM) and maintains the WebSocket connection to AosCloud. Secondary Nodes need a way to:

  • Receive desired-state commands from CM (service deployments, configuration updates)
  • Report status, monitoring data, and alerts back to CM
  • Obtain service images for local deployment
  • Forward logs to the cloud
  • Access IAM services on the Main Node for certificate operations

The Message Proxy fulfills all of these needs by acting as a relay between the Secondary Node's local components and the Main Node's CM and IAM services.

Process: aos_messageproxy

Architecture​

The MP establishes a transport connection to the Main Node and multiplexes multiple logical channels over it. Each channel carries a different type of traffic (CM messages, IAM public API, IAM protected API), routed by port number in a custom protocol header.

┌─────────────────────────────────────────────────────────┐
│                   Secondary Node                         │
│                                                         │
│  ┌──────────┐    ┌──────────────────────────────────┐   │
│  │ CMClient │◄──►│       Communication Manager       │   │
│  │  (gRPC)  │    │  ┌────────────┐ ┌─────────────┐  │   │
│  └──────────┘    │  │ CM Channel │ │ IAM Channels │  │   │
│                  │  └────────────┘ └─────────────┘  │   │
│  ┌──────────┐    │         │               │         │   │
│  │PublicNode│◄──►│    ┌────┴───────────────┴────┐    │   │
│  │ Client   │    │    │   Transport (vchan/TCP)  │    │   │
│  └──────────┘    └────┼─────────────────────────┼────┘   │
│                       │                         │         │
└───────────────────────┼─────────────────────────┼─────────┘
                        │    Inter-Node Link       │
┌───────────────────────┼─────────────────────────┼─────────┐
│                       ▼                         ▼         │
│                  Main Node (CM + IAM)                      │
└───────────────────────────────────────────────────────────┘

Subcomponents​

ModuleResponsibility
cmclientgRPC client connecting to the Main Node's CM — registers the Secondary Node's SM, relays commands and status using the servicemanager v4 proto
communicationManages the inter-node transport layer — multiplexes logical channels over a single transport connection, handles connection lifecycle and message framing
filechunkerSplits large files (service images) into chunks for reliable transfer across the inter-node transport
imageunpackerUnpacks received OCI service image archives for consumption by the local Service Manager
logproviderArchives and forwards logs — collects log data from the local Node and transmits it to CM via the inter-node transport
iamclientgRPC client connecting to the Main Node's IAM — provides the Secondary Node with access to IAM public and protected APIs for certificate and Node management operations
configMP process configuration loading and validation

Communication Layer​

Transport Backends​

The MP supports two transport backends, selected at build time:

BackendBuild FlagUse Case
Xen vchanWITH_VCHAN=ONXen hypervisor environments where Nodes are VMs on the same host — uses Xen virtual channels for low-latency inter-VM communication
TCP socketWITH_VCHAN=OFF (default)All other configurations — uses a TCP server socket that the Main Node connects to

Both backends implement the same TransportItf interface, making the transport selection transparent to the rest of the MP.

Channel Multiplexing​

The Communication Manager module multiplexes multiple logical channels over the single transport connection. Each message is framed with a protocol header containing:

  • Port number — identifies which logical channel the message belongs to
  • Data size — length of the payload
  • SHA-256 checksum — integrity verification of the payload

This allows the MP to carry CM traffic, IAM public API traffic, and IAM protected API traffic simultaneously over one transport link.

Secure Channels​

Logical channels can be established as either open (unencrypted) or secure (TLS-encrypted):

  • Open channels — used during provisioning mode when certificates are not yet available
  • Secure channels — used in normal operation; wrap the underlying communication channel with TLS using OpenSSL, authenticated with IAM-issued certificates

The MP subscribes to certificate change notifications from IAM. When certificates are rotated, the transport connection is closed and re-established with the new credentials.

CM Client​

The cmclient module is the MP's interface to the Main Node's Communication Manager. It:

  1. Establishes a gRPC connection to CM's SMService (using the servicemanager v4 proto)
  2. Registers the Secondary Node's SM with CM
  3. Relays incoming commands from CM to the local SM (via the inter-node transport)
  4. Forwards outgoing status, monitoring data, and alerts from the local SM back to CM

The CM client implements the HandlerItf interface, meaning it receives connection/disconnection notifications from the communication layer and handles message send/receive operations. It caches outgoing messages when disconnected and sends them once the connection is restored.

IAM Connectivity​

The MP provides the Secondary Node with access to the Main Node's IAM through two connections:

  • Public IAM connection — carries certificate requests and public API operations (open port, always available)
  • Protected IAM connection — carries provisioning and Node management operations (secure port, requires TLS)

Each IAM connection runs as an independent channel within the communication layer, with its own port assignment and optional TLS wrapping.

Image Distribution​

When the Main Node's CM assigns a service to a Secondary Node, the MP handles the image transfer:

  1. Download — the CM connection triggers a download of the service image from the configured URL
  2. Chunking — the filechunker splits the downloaded image into chunks, computing SHA-256 checksums for each file
  3. Transfer — chunks are sent over the inter-node transport to the Main Node (or received from it, depending on the flow)
  4. Unpacking — the imageunpacker extracts the OCI image archive into the local image store directory for the SM to consume

Log Forwarding​

The logprovider module (via ArchiveManager) handles log data received from the local Node:

  1. Receives log data chunks from the SM
  2. Archives log entries using a configurable archivator
  3. Forwards completed log archives to CM through the CM connection
  4. Manages cleanup of outdated archives on a periodic timer

Provisioning Mode​

The MP supports a provisioning mode (activated with the --provisioning flag) for initial Node setup:

  • In provisioning mode, connections are established without TLS (insecure)
  • The protected IAM connection is not initialized
  • Certificate subscriptions are skipped

This allows the MP to operate before the Node has obtained its initial certificates from IAM.

Configuration​

The MP is configured through a JSON configuration file (aos_message_proxy.cfg). Key configuration sections:

SectionPurpose
CMConfigCM server URL and port assignments for open/secure channels
VChanConfigXen vchan domain ID and XenStore paths (when using vchan transport)
IAMConfigIAM server URLs (local public, Main Node public, Main Node protected) and port assignments
DownloadDownload directory, concurrency limits, and retry parameters
CertStorageCertificate storage identifier for TLS credential management
ImageStoreDirLocal directory for unpacked service images

Build Configuration​

The Message Proxy is included in the build with:

-DWITH_MP=ON

The transport backend is selected with:

-DWITH_VCHAN=ON   # Use Xen vchan transport
-DWITH_VCHAN=OFF  # Use TCP socket transport (default)

A typical Secondary Node image is built with WITH_SM=ON, WITH_IAM=ON, WITH_MP=ON, and WITH_CM=OFF.