Storage State Management

Introduction

Each service instance running on a Node can have two types of persistent data: storage and state. These serve different purposes and have different lifecycle behaviors. Understanding the distinction is essential for OEMs developing services that need to persist data across restarts or version upgrades.

The storage state management system is implemented in the Communication Manager (CM) and enforced by the Service Manager (SM) at runtime. CM manages the allocation, quota enforcement, and cloud synchronization of storage and state, while SM mounts the appropriate directories into the service container at launch time.

Storage vs State

AosCore distinguishes between two persistent data categories for each service instance:

Property	Storage	State
Purpose	General-purpose writable storage for the service instance	Small persistent state that survives service version updates
Container mount point	/storage (directory)	/state.dat (single file)
Survives restarts	Yes	Yes
Survives version updates	No — cleared when the service is removed and re-deployed at a new version	Yes — preserved across version upgrades of the same service
Survives instance removal	No — removed when the instance is permanently removed from the desired state	No — removed when the instance is permanently removed
Cloud synchronization	No	Yes — state changes are reported to AosCloud and can be updated from the cloud
Typical use	Caches, temporary databases, downloaded assets	Configuration state, license data, calibration values

Directory Structure

The CM maintains two top-level directories for storage and state, configured via the storageDir and stateDir fields in the storage state configuration:

<storageDir>/
└── <itemID>/
    └── <subjectID>/
        └── <instanceIndex>/     ← writable storage directory for this instance

<stateDir>/
└── <itemID>/
    └── <subjectID>/
        └── <instanceIndex>/
            └── state.dat        ← state file for this instance

Each instance is identified by the combination of itemID, subjectID, and instanceIndex (the InstanceIdent triple). This ensures complete isolation between instances — even multiple instances of the same service have separate storage and state directories.

Setup and Lifecycle

When CM schedules an instance to a Node, it calls the StorageState::Setup method to prepare the storage and state for that instance. The setup process:

1. Creates or retrieves metadata — checks the persistent database for existing storage state info for this instance. If none exists (first deployment), creates a new record.
2. Prepares storage — creates the storage directory if the storage quota is non-zero, sets ownership to the instance's UID/GID, and returns the relative storage path.
3. Prepares state — creates the state directory and state.dat file if the state quota is non-zero, sets ownership, starts filesystem watching for change detection, and returns the relative state path.
4. Enforces quotas — applies filesystem user quotas to limit how much disk space the instance can consume.

The resulting storage_path and state_path are included in the InstanceInfo message sent to the SM via the UpdateInstances gRPC call.

Quota Enforcement

Quotas are enforced at the filesystem level using per-user disk quotas. Each instance runs under a unique UID, and the quota is applied to that UID on the relevant partition:

• If storage and state reside on the same partition, a single combined quota (storageQuota + stateQuota) is applied.
• If they reside on different partitions, separate quotas are applied to each partition independently.

The system detects whether storage and state share a partition by comparing their mount points at initialization time.

Container Mount Behavior

When the SM launches a container instance, it mounts the storage and state into the container's filesystem:

Host path	Container path	Mount type
<storageDir>/<relative_storage_path>	/storage	bind mount, read-write
<stateDir>/<relative_state_path>/state.dat	/state.dat	bind mount, read-write

The service process inside the container accesses:

• /storage — a writable directory for general-purpose persistent data
• /state.dat — a writable file for small persistent state

If either path is empty (quota is zero), the corresponding mount is not created and the service does not have access to that storage type.

State Synchronization with Cloud

The state file (state.dat) has special cloud synchronization behavior that storage does not:

Change Detection

The CM watches the state file for changes using a filesystem event watcher (FSWatcherItf). When a change is detected:

1. The file content is read
2. A SHA3-224 checksum is calculated
3. If the checksum differs from the last known checksum, the new state is sent to AosCloud via SendNewState

This allows the cloud to maintain an up-to-date copy of each instance's state.

Cloud-Initiated State Updates

The cloud can push state updates to an instance via the UpdateState message:

1. CM receives the update with the new state content and checksum
2. CM validates the checksum matches the content
3. CM verifies the state size does not exceed the instance's state quota
4. CM writes the new content to the instance's state.dat file
5. CM updates the stored checksum in the database

State Acceptance Protocol

After a state change (either local or cloud-initiated), the cloud can accept or reject the state:

• Accepted — the checksum is persisted as the confirmed state
• Rejected — CM sends a StateRequest to retrieve the current state from the instance, triggering a re-synchronization

This protocol ensures that the cloud and the instance maintain a consistent view of the state, even across network interruptions.

Behavior Across Updates

When a service is updated to a new version:

1. The old instance is stopped (removed from the desired state)
2. The new instance is started with the same InstanceIdent (same itemID, subjectID, instanceIndex)
3. Because the state directory path is derived from the InstanceIdent, the state file persists — the new version inherits the previous version's state
4. Storage behavior depends on whether the instance identity changes — if the same InstanceIdent is reused, storage also persists

Instance Removal

When an instance is permanently removed (no longer in the desired state):

1. StorageState::Remove is called
2. Filesystem watching for the state file is stopped
3. Both the state directory and storage directory are deleted from the filesystem
4. The metadata record is removed from the persistent database

Instance Cleanup

When an instance is temporarily stopped (e.g., during rebalancing):

1. StorageState::Cleanup is called
2. Filesystem watching is stopped (no more change notifications)
3. The storage and state directories are not deleted — they remain on disk for when the instance is restarted

Configuration

The storage state system is configured through the CM configuration file:

Field	Description	Default
storageDir	Base directory for instance storage	(required)
stateDir	Base directory for instance state files	(required)

On the SM side, the container runtime has corresponding configuration:

Field	Description	Default
storageDir	Directory where SM expects storage mounts	<workingDir>/storages
stateDir	Directory where SM expects state mounts	<workingDir>/states

Related Pages

• Service Lifecycle — overview of the complete service lifecycle including storage context
• Service Instance States — how instance state transitions interact with storage setup and cleanup

• SM Launcher — how the Service Manager launches containers with mounted storage
• SM Resource Manager — resource allocation including storage quotas
• Storage and Quota Configuration — configuration reference for storage and quota settings