Skip to main content

Getting logs and service instances status

Introduction​

Upon establishing the SSH connection to the designated unit, users can access comprehensive service instance logs alongside the system logs. Additionally, detailed information regarding the status of each instance can be retrieved. This functionality allows efficient monitoring and troubleshooting of service performance and system health, ensuring optimal operational integrity.

Aos service instances are deployed and administrated as systemd services utilizing the crun container runtime. Consequently, users can leverage standard systemd and crun utilities to efficiently obtain the status and logs of any service instance.

Before initiating a service instance, the Aos service manager assigns an instance ID to each instance. This instance ID serves as a unique identifier, enabling management of the service instance throughout its lifecycle.

Getting service instance status​

Getting systemd unit status of service instance​

To retrieve all Aos service instances running on the particular node, systemctl command can be used:

systemctl | grep aos-service@

It displays systemd unit statuses corresponding to all Aos service instances running on this node:

  aos-service@a524897a-fdc3-4bcd-a81f-6fa136f16e3b.service                                 loaded active running   AOS Service
  aos-service@ecd435e5-06cb-43bb-a3cf-ee58e14230a4.service                                 loaded active running   AOS Service

To get more details about specific instance, issue systemctl status command:

systemctl status {UNIT}

Where {UNIT} is systemd unit name displayed above. It has the following format: aos-service@{INSTANCEID}.service.

For example:

systemctl status aos-service@ecd435e5-06cb-43bb-a3cf-ee58e14230a4.service

● aos-service@ecd435e5-06cb-43bb-a3cf-ee58e14230a4.service - AOS Service
     Loaded: loaded (/usr/lib/systemd/system/aos-service@.service; disabled; preset: disabled)
    Drop-In: /run/systemd/system/aos-service@ecd435e5-06cb-43bb-a3cf-ee58e14230a4.service.d
             └─parameters.conf
     Active: active (running) since Tue 2024-12-10 14:53:14 UTC; 45min ago
    Process: 1184 ExecStartPre=/usr/bin/crun delete -f ecd435e5-06cb-43bb-a3cf-ee58e14230a4 (code=exited, status=0/SUCCESS)
    Process: 1199 ExecStart=/usr/bin/crun run -d --pid-file /run/aos/runtime/ecd435e5-06cb-43bb-a3cf-ee58e14230a4/.pid -b /run/aos/runtime/ecd435e5-06cb-43bb-a3cf-ee58e14230a4 ecd435e5-06cb-43bb-a3cf-ee58e14230a4 (code=exited, status=0/SUCCESS)
   Main PID: 1203 (python3)
      Tasks: 0 (limit: 2284)
     Memory: 724.0K (peak: 1.5M)
        CPU: 6ms
     CGroup: /system.slice/system-aos\x2dservice.slice/aos-service@ecd435e5-06cb-43bb-a3cf-ee58e14230a4.service
             ‣ 1203 python3 -u main.py

Dec 10 14:53:14 main systemd[1]: Starting AOS Service...
Dec 10 14:53:14 main systemd[1]: Started AOS Service.

Getting container status of service instance​

To get status of container, corresponding to Aos service instance, crun command line could be used. To obtain the list of running containers, the following command should be executed:

crun --root=/run/crun list

This command provides a detailed overview of currently running containers along with their statuses, as illustrated below:

NAME                                 PID       STATUS   BUNDLE PATH                             CREATED                        OWNER
a524897a-fdc3-4bcd-a81f-6fa136f16e3b 1209      running  /run/aos/runtime/a524897a-fdc3-4bcd-a81f-6fa136f16e3b 2024-12-10T14:53:15.000615Z    root
ecd435e5-06cb-43bb-a3cf-ee58e14230a4 1203      running  /run/aos/runtime/ecd435e5-06cb-43bb-a3cf-ee58e14230a4 2024-12-10T14:53:14.987746Z    root

In above example, container name is instance ID.

To retrieve more detailed status about specific container, the following command should be used:

crun --root=/run/crun state {INSTANCEID}

For example:

crun --root=/run/crun state a524897a-fdc3-4bcd-a81f-6fa136f16e3b

{
    "ociVersion": "1.0.0",
    "id": "a524897a-fdc3-4bcd-a81f-6fa136f16e3b",
    "pid": 1209,
    "status": "running",
    "bundle": "/run/aos/runtime/a524897a-fdc3-4bcd-a81f-6fa136f16e3b",
    "rootfs": "/run/aos/runtime/a524897a-fdc3-4bcd-a81f-6fa136f16e3b/rootfs",
    "created": "2024-12-10T14:53:15.000615Z",
    "systemd-scope": "",
    "owner": "root"
}

Identifying service instance​

Each service instance is initialized with predefined environment variables encapsulating identity information. These variables provide the following metadata: the service ID, subject ID and instance index. To get value of these variable, env command should be performed inside the container as following:

crun --root=/run/crun exec {INSTANCEID} env | grep AOS_

For example:

crun --root=/run/crun exec a524897a-fdc3-4bcd-a81f-6fa136f16e3b env | grep AOS_

AOS_SERVICE_ID=ee36b86f-69d3-4376-9895-1a5618eb6e3e
AOS_SUBJECT_ID=ff7a8084-3545-4de8-84d4-fba02c03111c
AOS_INSTANCE_INDEX=0
AOS_INSTANCE_ID=a524897a-fdc3-4bcd-a81f-6fa136f16e3b

As seen in above example, instance a524897a-fdc3-4bcd-a81f-6fa136f16e3b is instance of service ee36b86f-69d3-4376-9895-1a5618eb6e3e and subject ff7a8084-3545-4de8-84d4-fba02c03111c.

Running commands inside service instance container​

The exec command of the crun runtime allows users to execute any command available within the container for debugging or inspection purposes. This functionality provides a powerful tool for administrators to interact with the containerized environment, enabling the execution of diagnostic queries, configuration checks, or other investigative tasks directly within the running container.

crun --root=/run/crun exec {INSTANCEID} {COMMAND}

For example, executing bash inside specific container of Aos service instance:

crun --root=/run/crun exec a524897a-fdc3-4bcd-a81f-6fa136f16e3b bash

Getting system and service instances log​

Getting system logs​

All logs are available in systemd journal and can be inspected with journalctl command:

journalctl

or

journalctl -f

to see the most recent journal entries.

Getting service instances log​

To obtain systemd unit log of Aos service instance the systemd unit should be specified:

journalctl -u {UNIT}

For example:

journalctl -u aos-service@ecd435e5-06cb-43bb-a3cf-ee58e14230a4.service

This command displays only systemd unit log. To get the output produced by Aos instance inside container, journal logs should be filtered by croup path:

journalctl _SYSTEMD_CGROUP=/system.slice/system-aos\\x2dservice.slice/{INSTANCEID}

For example:

journalctl _SYSTEMD_CGROUP=/system.slice/system-aos\\x2dservice.slice/ecd435e5-06cb-43bb-a3cf-ee58e14230a4

"-f" option could be used to display the most recent service log:

journalctl _SYSTEMD_CGROUP=/system.slice/system-aos\\x2dservice.slice/ecd435e5-06cb-43bb-a3cf-ee58e14230a4 -f

CGroup path of the specific instance could be acquired from the container runtime spec, using the following command:

cat /run/aos/runtime/{INSTANCEID}/config.json | grep cgroupsPath

Example:

cat /run/aos/runtime/ecd435e5-06cb-43bb-a3cf-ee58e14230a4/config.json | grep cgroupsPath

"cgroupsPath": "/system.slice/system-aos\\x2dservice.slice/ecd435e5-06cb-43bb-a3cf-ee58e14230a4"