Version: v1.1

Multi-Node Architecture

Introduction

This page describes the internal architecture that enables multi-Node operation in AosEdge Units. It covers the physical and logical topology, the transport layer that connects Nodes, the channel multiplexing protocol that carries multiple traffic types over a single link, and how the Service Manager and IAM connections are established across Node boundaries.

The multi-Node architecture is an edge-side implementation detail — the cloud communicates with the Unit as a single entity and has no awareness of the Message Proxy or inter-Node transport. From the OEM integration perspective, understanding this architecture helps with:

Selecting the appropriate transport backend for your hardware topology
Configuring port assignments and channel security
Diagnosing inter-Node connectivity issues
Understanding how service deployments and certificate operations reach Secondary Nodes

Topology

A multi-Node Unit follows a star topology with the Main Node at the center:

                        ┌─────────────────────────────────┐
                        │           AosCloud              │
                        └────────────────┬────────────────┘
                                         │ WebSocket (JSON)
                                         │
                        ┌────────────────┴────────────────┐
                        │          Main Node               │
                        │                                  │
                        │   ┌────┐   ┌────┐   ┌─────┐     │
                        │   │ CM │   │ SM │   │ IAM │     │
                        │   └──┬─┘   └────┘   └──┬──┘     │
                        │      │                  │        │
                        └──────┼──────────────────┼────────┘
                               │                  │
              ┌────────────────┼──────────────────┼────────────────┐
              │                │                  │                │
    ┌─────────┴─────────┐     │     ┌────────────┴────────────┐   │
    │  Secondary Node A  │     │     │   Secondary Node B       │   │
    │                    │     │     │                          │   │
    │  ┌────┐  ┌─────┐  │     │     │  ┌────┐  ┌─────┐        │   │
    │  │ SM │  │ IAM │  │     │     │  │ SM │  │ IAM │        │   │
    │  └──┬─┘  └──┬──┘  │     │     │  └──┬─┘  └──┬──┘        │   │
    │     │       │      │     │     │     │       │            │   │
    │  ┌──┴───────┴──┐   │     │     │  ┌──┴───────┴──┐        │   │
    │  │     MP      │   │     │     │  │     MP      │        │   │
    │  └──────┬──────┘   │     │     │  └──────┬──────┘        │   │
    └─────────┼──────────┘     │     └─────────┼───────────────┘   │
              │                │               │                   │
              └────────────────┴───────────────┘                   │
                     Transport (vchan or TCP)                       │

Key characteristics:

Single cloud connection — only the Main Node's CM maintains the WebSocket connection to AosCloud
Star topology — each Secondary Node connects independently to the Main Node; Secondary Nodes do not communicate with each other
Per-Node identity — each Node (Main and Secondary) runs its own IAM and maintains its own certificates
Centralized orchestration — CM on the Main Node distributes desired state, collects status, and coordinates updates across all Nodes

Transport Layer

The transport layer provides the raw byte-stream connection between a Secondary Node and the Main Node. Two backends are available, selected at build time:

Xen Virtual Channel (vchan)

Used when Nodes are Xen virtual machines running on the same physical host.

Property	Value
Build flag	`WITH_VCHAN=ON`
Addressing	Xen domain ID + XenStore paths
Channels	Separate read and write vchan instances
Latency	Very low (shared-memory based)
Use case	Hypervisor-based multi-Node Units

The vchan transport uses the libxenvchan library. Each direction (read/write) is a separate virtual channel, addressed by the peer domain's ID and a XenStore path. The Secondary Node's MP creates the vchan endpoints and waits for the Main Node to connect.

Configuration:

{
  "VChan": {
    "Domain": 0,
    "XSRXPath": "/local/domain/1/data/vchan_rx",
    "XSTXPath": "/local/domain/1/data/vchan_tx",
    "IAMCertStorage": "iam",
    "SMCertStorage": "sm"
  }
}

TCP Socket

Used when Nodes are separate physical boards connected over a network, or any non-Xen configuration.

Property	Value
Build flag	`WITH_VCHAN=OFF` (default)
Addressing	TCP port (default: 30001)
Connection model	Server socket on Secondary Node; Main Node connects as client
Library	Poco::Net with reactor pattern
Use case	Networked boards, development environments

The TCP socket transport runs a server socket on the Secondary Node. The Main Node initiates the TCP connection to the Secondary Node's listening port. A Poco SocketReactor handles connection acceptance asynchronously.

Connection direction: The Main Node connects to the Secondary Node (the Secondary Node listens). This may seem counterintuitive, but it allows the Secondary Node to be ready before the Main Node starts, simplifying startup ordering.

Transport Interface

Both backends implement the same TransportItf interface:

Method	Purpose
`Connect()`	Establish the transport connection (blocks until peer connects)
`Read(message)`	Read raw bytes from the transport
`Write(message)`	Write raw bytes to the transport
`Close()`	Close the current connection (can reconnect)
`Shutdown()`	Permanently shut down the transport

This abstraction makes the rest of the MP stack transport-agnostic — the channel multiplexing, message framing, and TLS wrapping work identically regardless of whether the underlying link is vchan or TCP.

Channel Multiplexing

A single transport connection carries multiple logical channels simultaneously. Each channel is identified by a port number and carries a different type of traffic.

Protocol Header

Every message sent over the transport is prefixed with an AosProtocolHeader:

Field	Type	Size	Purpose
`mPort`	`uint32_t`	4 bytes	Identifies the destination logical channel
`mDataSize`	`uint32_t`	4 bytes	Length of the payload in bytes
`mCheckSum`	`uint8_t[32]`	32 bytes	SHA-256 checksum of the payload

The Communication Manager reads the header, validates the checksum, looks up the channel by port number, and delivers the payload to the correct channel's receive buffer.

Maximum message size: 64 KB. Messages exceeding this limit are rejected.

Logical Channels

The MP establishes the following logical channels over the single transport:

Channel	Port Source	Security	Purpose
CM Open	`CMConfig.mOpenPort`	Unencrypted	CM communication during provisioning
CM Secure	`CMConfig.mSecurePort`	TLS-encrypted	CM communication in normal operation
IAM Public	`IAMConfig.mOpenPort`	Unencrypted	IAM public API (certificate requests, always available)
IAM Protected	`IAMConfig.mSecurePort`	TLS-encrypted	IAM provisioning and Node management operations

In normal (non-provisioning) operation, the CM Secure and IAM Protected channels carry the primary traffic. The IAM Public channel remains available for certificate-related operations that must work before TLS is fully established.

Channel Security

Each logical channel can be either open (plaintext) or secure (TLS-encrypted):

Open channels — raw protobuf messages over the multiplexed transport. Used during provisioning when certificates are not yet available, and for the IAM public API.
Secure channels — wrap the underlying communication channel with TLS using OpenSSL. The SecureChannel class implements a custom BIO (Basic I/O) that reads/writes through the multiplexed channel rather than a raw socket. Authentication uses IAM-issued certificates.

When certificates are rotated (detected via IAM subscription notifications), the transport connection is closed and re-established with the new credentials. This triggers all channels to reconnect.

SM Connection Across Nodes

The Service Manager on each Node connects to the Main Node's CM via gRPC using the servicemanager v5 proto (RegisterSM RPC). The connection path differs by Node type:

Main Node SM

On the Main Node, SM connects directly to CM's gRPC server (localhost). No MP involvement.

Main Node:  SM ──gRPC──► CM (SMController)

Secondary Node SM

On a Secondary Node, SM connects to the local MP's CMClient, which relays messages through the inter-Node transport to the Main Node's CM:

Secondary Node:  SM ──gRPC──► MP (CMClient) ──[transport]──► Main Node CM (SMController)

The CMClient module within MP:

Establishes a gRPC connection to CM's SMService on the Main Node
Registers the Secondary Node's SM with CM (the RegisterSM bidirectional stream)
Relays incoming commands from CM (service deployments, configuration updates) to the local SM
Forwards outgoing status, monitoring data, alerts, and logs from the local SM back to CM
Caches outgoing messages when disconnected and sends them once the connection is restored

From CM's perspective, all SM connections look identical — it does not distinguish between a local SM and a remote SM connected via MP. The SM Controller manages all connected SMs uniformly.

IAM Registration Across Nodes

Each Secondary Node's IAM registers with the Main Node's IAM through the RegisterNode bidirectional gRPC stream (defined in IAMPublicNodesService). This registration is carried over the MP's IAM channels.

Registration Flow

Secondary Node's IAM establishes a RegisterNode stream to the Main Node's IAM
The stream carries IAMOutgoingMessages (Secondary → Main) and IAMIncomingMessages (Main → Secondary)
The Secondary Node sends its NodeInfo (node ID, type, capabilities, state) as the first outgoing message
The Main Node's IAM tracks the Secondary Node's state and makes it visible to the cloud

Traffic Carried Over IAM Channels

IAM Public channel (open port):

Certificate requests (GetCert)
Certificate change subscriptions (SubscribeCertChanged)
Node info queries

IAM Protected channel (secure port):

Provisioning operations (StartProvisioning, FinishProvisioning, Deprovision)
Certificate management (CreateKey, ApplyCert)
Node control (PauseNode, ResumeNode)

The two IAM connections (IAMPublicConnection and IAMProtectedConnection) run as independent channels within the communication layer, each with its own port assignment and optional TLS wrapping.

Connection Lifecycle

Startup Sequence

Transport initialization — MP initializes the transport backend (vchan or TCP socket)
Communication Manager start — begins the connection loop (connect, read, reconnect on failure)
CM Connection start — establishes open and secure CM channels, begins relaying messages
IAM Public Connection start — establishes the IAM public channel (always available)
IAM Protected Connection start — establishes the IAM protected channel (skipped in provisioning mode)

Reconnection Behavior

The Communication Manager runs a persistent connection loop:

Attempt to connect the transport (blocks until peer is available)
Enter the read loop — dispatch incoming messages to channels by port
On transport error — close all channels, wait 3 seconds, retry from step 1

Individual connections (CM, IAM) also handle disconnection gracefully:

CMClient caches outgoing messages during disconnection
IAM connections retry with a 3-second timeout between attempts
Certificate rotation triggers a full transport reconnect

Provisioning Mode

When MP starts with the --provisioning flag:

All connections use open (unencrypted) channels
The IAM Protected connection is not initialized
Certificate subscriptions are skipped
This allows the Node to operate before obtaining its initial certificates

Once provisioning completes and certificates are issued, MP is restarted in normal mode with full TLS security.

Message Flow Example

To illustrate how a service deployment reaches a Secondary Node:

Cloud → CM: AosCloud sends a desired state update over WebSocket to the Main Node's CM
CM → SM (via MP): CM determines the target Node and sends a RunInstances command through the SM Controller's gRPC stream
MP relay: On the Secondary Node, MP's CMClient receives the command over the secure CM channel and forwards it to the local SM via gRPC
SM execution: The local SM pulls the service image (via MP's file distribution), unpacks it, and starts the service
Status return: SM reports instance status back through the same path (SM → MP CMClient → transport → CM → cloud)

The entire flow is transparent — CM treats all SMs uniformly, and the cloud sees a single Unit status regardless of which Node runs which service.

Build Configuration

The transport backend is selected at build time:

# Xen vchan transport (for hypervisor environments)
-DWITH_VCHAN=ON

# TCP socket transport (default, for networked boards)
-DWITH_VCHAN=OFF

A typical Secondary Node build includes:

-DWITH_SM=ON    # Local service management
-DWITH_IAM=ON   # Per-Node identity
-DWITH_MP=ON    # Inter-Node communication
-DWITH_CM=OFF   # No cloud connection (handled by Main Node)

Multi-Node Operation — overview of multi-Node concepts and the Main Node model
Message Proxy — detailed MP component documentation including all submodules
Node Lifecycle — Node state machine and registration flow
Inter-Node File Distribution — how service images are transferred between Nodes
Architecture Overview — system-wide component architecture
Communication Manager — the Main Node component that coordinates all Nodes

Introduction​

Topology​

Transport Layer​

Xen Virtual Channel (vchan)​

TCP Socket​

Transport Interface​

Channel Multiplexing​

Protocol Header​

Logical Channels​

Channel Security​

SM Connection Across Nodes​

Main Node SM​

Secondary Node SM​

IAM Registration Across Nodes​

Registration Flow​

Traffic Carried Over IAM Channels​

Connection Lifecycle​

Startup Sequence​

Reconnection Behavior​

Provisioning Mode​

Message Flow Example​

Build Configuration​

Related Pages​