Security for AOS Edge

Privileges required by AOS Edge

The Aos Edge for AWS solution was designed to follow the least privilege principle for all the Aos Cloud services which will be installed on top of EKS.

Root privileges disclaimer

The AOS Edge application does NOT require root privileges, each pod has a role which it restrict the access only to certain resources depending on the application functionality.

To be able to deploy the AOS Edge in the AWS cloud root privileges are NOT needed either.

Privileges per AWS solution

• Secrets Manager /specific path and Cert Manager aos cert
  • app backend
  • celery - rabbitmq scaler
  • service discovery
  • alert handler
  • frontend nginx
  • grafana
  • velero
• S3 and CloudFront manipulation
  • app backend
  • service discover
  • velero
• EC2 manipulation
  • celery - rabbitmq scaler
    • EC2 - CloudTemplate, launch, destroy describe, only resources which are part of the cloudtemplate, and have tags which are related to this service, IAM it is configured to not allow manipulation out of the launch template and tags conditions.
• ECR
  • Read access to AKS role to get the needed images.
• EC2 LoadBalancer
  • Istio ingress controller (gateway)

IAM policies per Aos Cloud applications

Service / Role	Policies
app backend / <project>-<env>-app	Secret appcfg/appsec (custom policy) Secret dbcfg/dbsec (custom policy) Secret landing (custom policy) Secret mhcfg/mhsec (custom policy) S3 Read/Write backend (custom policy) KMS Read specific key (custom policy)
backup / <project>-<env>-backup	S3 Infra RWX and EC2 volume/snapshots RW (custom policy) KMS Read specific key (custom policy)
infra operations / <project>-<env>-base csi secrets provider	Secret base (custom policy) Secret dbcfg/dbsec (custom policy) Secret data services (custom policy) Secret landing (custom policy) Secret ingress (custom policy) KMS Read specific key (custom policy)
Cloudtrail / <project>-<env>-ct-cloudwatch-role Needed for cloudwatch events attached to cloudtrail (CDN)	AWS Service: cloudtrail
data services / <project>-<env>-data-services	Secret data services (custom policy) KMS Read specific key (custom policy)
<project>-<env>-ecr-sync-lambda-role	Secret manager RW (custom policy) AWSXrayWriteOnlyAccess AmazonEC2ContainerRegistryFullAccess AWSLambdaExecute AWSLambdaENIManagementAccess
EKS <project>-<env>-eks-autoscaling-role <project>-<env>-eks-load-balancer-controller <project>-<env>-eks-node-role <project>-<env>-eks-role <project>-<env>-eks-ebs-controller <project>-<env>-eks-kcmpatch-lambda-lambda-role	Each role has least privilege principle to manage only resources needed Cluster Autoscaler (custom policy) EC2 EBA management (Custom policy) EC2 LB Management (Custom policy) EC2 Autoscaling management (custom policy) AWS service eks AWS service ec2 AWSLambdaExecute AWSLambdaENIManagementAccess EKS IdP
<project>-<env>-grafana	Secret grafana (custom policy) KMS Read specific key (custom policy)
<project>-<env>-helm-install-lambda-role	AWS Service: lambda AWS service eks EKS IdP
<project>-<env>-sd	Secret sdcfg/sdsec (custom policy) KMS Read specific key (custom policy)
<project>-<env>-secrets-manager	Secrets Manager (custom policy) KMS Read specific key (custom policy)
<project>-<env>-task	Secret grafana (custom policy) EC2 Management (custom policy) KMS Read specific key (custom policy)
<project>-<env>-terraform-admin This is needed to deploy all the needed resources for the Aos Edge Project. Just few people should be able to assume this role.	AWS Administrator access

In continuation we will describe secrets which are being used and created by terraform.

Secrets and Keys

AWS Secrets Manager secrets

All the AOS Edge properties are being saved as secrets, these can be found in the secrets.tf file when deployment it is acchieved, each secret it is part of a secret path which we call, we had decided to use this approach to split permissions between roles and pods/services, these will be setup in AWS Secrets manager.

In summary we have the following secrets which will be provisioned and managed by terraform:

<project>-<env>-appsec/AppCloudSecret
<project>-<env>-jobsec/JobCloudUser
<project>-<env>-jobsec/JobCloudSecret
<project>-<env>-base/SiteDomain
<project>-<env>-jobcfg/SiteDomain
<project>-<env>-appcfg/MaxComponentInstallAttempts
<project>-<env>-jobcfg/MaxComponentInstallAttempts
<project>-<env>-base/AosRootCertificatePem
<project>-<env>-appsec/AosRootCertificatePem
<project>-<env>-jobsec/AosRootCertificatePem
<project>-<env>-sdsec/AosRootCertificatePem
<project>-<env>-appsec/AosOemCaCertificatesPem
<project>-<env>-appsec/AosOemCaKeyPem
<project>-<env>-jobsec/AosMessageQueueTrustListPem
<project>-<env>-jobsec/AosInternalCACertificatesPem
<project>-<env>-jobsec/AosMessageQueueServerCertificatesPem
<project>-<env>-jobsec/AosMessageQueueManagementServerCertificatesPem
<project>-<env>-jobsec/AosMessageQueueClientCertificatesPem
<project>-<env>-jobsec/AosMessageQueueServerKeyPem
<project>-<env>-jobsec/AosMessageQueueManagementServerKeyPem
<project>-<env>-jobsec/AosMessageQueueClientKeyPem
<project>-<env>-jobsec/AosMessageHandlerCertificatesPem
<project>-<env>-jobsec/AosMessageHandlerKeyPem
<project>-<env>-custom-ingress/3internalCA_fullchain
<project>-<env>-infrastructure/ovpnBase64Config
<project>-<env>-custom-ingress/CertificateServer
<project>-<env>-custom-ingress/CertificateServerKey
<project>-<env>-base/RootCertificate
<project>-<env>-appsec/RootCertificate
<project>-<env>-jobsec/RootCertificate
<project>-<env>-sdsec/RootCertificate
<project>-<env>-ingress/RootCertificate
<project>-<env>-jobsec/UnitsAmqpTrustedCAs
<project>-<env>-jobsec/UnitsManagementTrustedCAs
<project>-<env>-jobsec/InternalCACertificatesPem
<project>-<env>-jobsec/MessageHandlerAppClientCertificates
<project>-<env>-jobsec/MessageHandlerAppClientKey
<project>-<env>-jobsec/MessageHandlerAmqpClientCertificates
<project>-<env>-jobsec/MessageHandlerAmqpClientKey
<project>-<env>-jobsec/UnitsAmqpServerCertificates
<project>-<env>-jobsec/UnitsAmqpServerKey
<project>-<env>-jobsec/UnitsManagementServerCertificates
<project>-<env>-jobsec/UnitsManagementServerKey
<project>-<env>-ingress/AppCertificates
<project>-<env>-ingress/AppKey
<project>-<env>-terraform-fwey-terraform
<project>-<env>-appcfg/ProjectName
<project>-<env>-appcfg/Environment
<project>-<env>-appcfg/AWSRegion
<project>-<env>-appcfg/AWSAccountId
<project>-<env>-appsec/InitialValue
<project>-<env>-appsec/EmailUseTls
<project>-<env>-grafana/GrafanaPassword
<project>-<env>-grafana/GrafanaUser
<project>-<env>-ingress/FakeCACertificate
<project>-<env>-ingress/FakeServerKey
<project>-<env>-ingress/FakeClientKey
<project>-<env>-ingress/FakeServerCertificate
<project>-<env>-ingress/FakeClientCertificate
<project>-<env>-ingress/FakeCAKey
<project>-<env>-sdsec/UnitsCacheRedisPassword
<project>-<env>-sdcfg/UnitsMessageQueueCloudExchangeName
<project>-<env>-sdcfg/UnitsCacheRedisQueueDb
<project>-<env>-sdcfg/UnitsCacheRedisConnectionTimeout
<project>-<env>-sdcfg/UnitsMessageQueueUnitsExchangeName
<project>-<env>-sdcfg/UnitsCacheRedisURL
<project>-<env>-sdcfg/UnitsCacheRedisSessionDb
<project>-<env>-sdcfg/RedisAttemptsNumberWhileKeyCollision
<project>-<env>-mhcfg/AppCacheRedisURL
<project>-<env>-taskcfg/InitailValue
<project>-<env>-taskcfg/ProjectName
<project>-<env>-dbcfg/PgBouncerURL
<project>-<env>-taskcfg/Environment
<project>-<env>-taskcfg/ContainerRegistryName
<project>-<env>-dbcfg/PostgresDbName
<project>-<env>-dbcfg/PostgresSchemaName
<project>-<env>-mhsec/AppCacheRedisPassword
<project>-<env>-mhsec/UnitsCacheRedisPassword
<project>-<env>-dbsec/PostgresEndpoint
<project>-<env>-dbsec/PostgresUserLogin
<project>-<env>-dbsec/PostgresUserPassword
<project>-<env>-base/SDSecretManagerSecret
<project>-<env>-base/AppSecretManagerUser
<project>-<env>-base/PgBouncerReservePoolTimeout
<project>-<env>-base/GrafanaPassword
<project>-<env>-base/ContainerRegistryUser
<project>-<env>-base/taskCloudUser
<project>-<env>-base/PgBouncerMaxClientConnections
<project>-<env>-base/UnitsCacheRedisURL
<project>-<env>-base/VeleroStorageSecret
<project>-<env>-base/GrafanaUser
<project>-<env>-base/AppCloudUser
<project>-<env>-base/Environment
<project>-<env>-base/PgBouncerReservePoolSize
<project>-<env>-base/PgBouncerMaxDbConnections
<project>-<env>-base/VeleroStorageUser
<project>-<env>-base/AppCloudSecret
<project>-<env>-base/SDConfigManagerURL
<project>-<env>-base/PgBouncerMinPoolSize
<project>-<env>-base/AppCacheRedisPassword
<project>-<env>-base/PostgresUserPassword
<project>-<env>-base/mhSecretManagerURL
<project>-<env>-base/ContainerRegistryName
<project>-<env>-base/taskConfigManagerURL
<project>-<env>-base/AppSecretManagerSecret
<project>-<env>-base/AppSecretManagerType
<project>-<env>-base/PgBouncerIniFile
<project>-<env>-base/PgBouncerDefaultPoolSize
<project>-<env>-base/SDSecretManagerUser
<project>-<env>-base/AppCacheRedisURL
<project>-<env>-base/mhConfigManagerURL
<project>-<env>-base/SDSecretManagerURL
<project>-<env>-base/taskSecretManagerType
<project>-<env>-base/PostgresAdminLogin
<project>-<env>-base/taskSecretManagerSecret
<project>-<env>-base/taskSecretManagerURL
<project>-<env>-base/taskSecretManagerUser
<project>-<env>-base/PgBouncerUsersFile
<project>-<env>-base/AppConfigManagerURL
<project>-<env>-base/PostgresSchemaName
<project>-<env>-base/DBConfigManagerURL
<project>-<env>-base/PostgresServerURL
<project>-<env>-base/AWSRedisURL
<project>-<env>-base/UnitsCacheRedisPassword
<project>-<env>-base/AWSAccountId
<project>-<env>-base/PostgresUserLogin
<project>-<env>-base/PostgresAdminPassword
<project>-<env>-base/taskCloudSecret
<project>-<env>-base/ContainerRegistryPassword
<project>-<env>-base/DBSecretManagerURL
<project>-<env>-base/SDSecretManagerType
<project>-<env>-base/ProjectName
<project>-<env>-base/PostgresDbName
<project>-<env>-base/AppSecretManagerURL
<project>-<env>-dbcfg/PostgresPort
<project>-<env>-tasksec/MessageHandlerCertificateChain
<project>-<env>-tasksec/ProtectUnitsDataCertificateChain
<project>-<env>-tasksec/UnitsAmqpTrustedCAs
<project>-<env>-ingress/IngressCACertificate
<project>-<env>-mhsec/MessageHandlerKey
<project>-<env>-tasksec/MessageHandlerAmqpClientCertificates
<project>-<env>-tasksec/UnitUpdatesCertificateChain
<project>-<env>-tasksec/MessageHandlerAppClientKey
<project>-<env>-tasksec/ProtectServerSecretsKey
<project>-<env>-tasksec/UnitsManagementTrustedCAs
<project>-<env>-tasksec/OemCAKey
<project>-<env>-tasksec/UnitQueuesManagementClientCertificateChain
<project>-<env>-tasksec/UnitCACertificateChain
<project>-<env>-tasksec/ServiceUpdatesCertificateChain
<project>-<env>-tasksec/UnitsAmqpServerKey
<project>-<env>-tasksec/ServiceUpdatesCertificate
<project>-<env>-tasksec/ProtectUnitsDataKey
<project>-<env>-tasksec/SecondaryCACertificate
<project>-<env>-tasksec/UnitCAKey
<project>-<env>-tasksec/UnitUpdatesKey
<project>-<env>-tasksec/ServiceProviderCACertificate
<project>-<env>-tasksec/MessageHandlerCertificate
<project>-<env>-tasksec/InternalCACertificate
<project>-<env>-tasksec/UnitUpdatesCertificate
<project>-<env>-tasksec/MessageHandlerAmqpClientKey
<project>-<env>-tasksec/MessageHandlerAppClientCertificates
<project>-<env>-tasksec/UnitCACertificate
<project>-<env>-tasksec/RootCACertificate
<project>-<env>-tasksec/ProtectUnitsDataCertificate
<project>-<env>-tasksec/UnitQueuesAmqpClientCertificateChain
<project>-<env>-tasksec/OemCACertificateChain
<project>-<env>-tasksec/UnitQueuesAmqpClientKey
<project>-<env>-tasksec/ServiceUpdatesKey
<project>-<env>-tasksec/RootCertificate
<project>-<env>-tasksec/MessageHandlerKey
<project>-<env>-tasksec/UnitsManagementServerCertificates
<project>-<env>-tasksec/UnitsManagementServerKey
<project>-<env>-tasksec/UnitQueuesManagementClientKey
<project>-<env>-tasksec/InvitationUserCertificate
<project>-<env>-tasksec/UnitsAmqpServerCertificates
<project>-<env>-tasksec/InternalCAKey
<project>-<env>-mhsec/UnitQueuesAmqpClientCertificateChain
<project>-<env>-tasksec/ServiceProviderCAKey
<project>-<env>-tasksec/InternalCACertificateChain
<project>-<env>-ingress/IngressCertificate
<project>-<env>-tasksec/OemCACertificate
<project>-<env>-mhsec/RootCACertificate
<project>-<env>-ingress/IngressKey
<project>-<env>-tasksec/ProtectServerSecretsCertificate
<project>-<env>-tasksec/ServiceProviderCACertificateChain
<project>-<env>-mhsec/UnitQueuesAmqpClientKey
<project>-<env>-mhsec/MessageHandlerCertificateChain
<project>-<env>-appcfg/AppCacheRedisURL
<project>-<env>-appcfg/DjangoMailerFromField
<project>-<env>-appcfg/SMTPMailServerPort
<project>-<env>-appcfg/SMTPMailServerHostname
<project>-<env>-appcfg/SupportedProvisioningSoft
<project>-<env>-appcfg/DjangoMailerErrorFromField
<project>-<env>-appsec/ServiceProviderCACertificate
<project>-<env>-appsec/UnitQueuesManagementClientCertificateChain
<project>-<env>-appsec/RootCACertificate
<project>-<env>-appsec/MessageHandlerCertificateChain
<project>-<env>-appsec/OemCACertificateChain
<project>-<env>-appsec/ProtectServerSecretsKey
<project>-<env>-appsec/ServiceProviderCACertificateChain
<project>-<env>-appsec/InternalCACertificateChain
<project>-<env>-appsec/OemCACertificate
<project>-<env>-appsec/ProtectUnitsDataKey
<project>-<env>-appsec/InternalCACertificate
<project>-<env>-appsec/UnitCACertificate
<project>-<env>-appsec/ProtectUnitsDataCertificateChain
<project>-<env>-appsec/ServiceUpdatesCertificate
<project>-<env>-appsec/OemCAKey
<project>-<env>-appsec/InternalCAKey
<project>-<env>-appsec/ProtectUnitsDataCertificate
<project>-<env>-appsec/UnitUpdatesCertificate
<project>-<env>-appsec/ServiceProviderCAKey
<project>-<env>-appsec/UnitQueuesManagementClientKey
<project>-<env>-appsec/ServiceUpdatesCertificateChain
<project>-<env>-appsec/UnitCACertificateChain
<project>-<env>-appsec/UnitUpdatesKey
<project>-<env>-appsec/UnitCAKey
<project>-<env>-appsec/MessageHandlerKey
<project>-<env>-appsec/ServiceUpdatesKey
<project>-<env>-appsec/SecondaryCACertificate
<project>-<env>-appsec/UnitUpdatesCertificateChain
<project>-<env>-appsec/UnitQueuesAmqpClientCertificateChain
<project>-<env>-appsec/ProtectServerSecretsCertificate
<project>-<env>-appsec/MessageHandlerCertificate
<project>-<env>-appsec/UnitQueuesAmqpClientKey
<project>-<env>-appsec/DjangoSecretKey
<project>-<env>-appsec/SMTPMailServerLogin
<project>-<env>-appsec/SMTPMailServerPassword
<project>-<env>-appsec/AppCacheRedisPassword
<project>-<env>-appcfg/SiteDomain
<project>-<env>-appcfg/UnitsCacheRedisURL
<project>-<env>-appsec/UnitsQueuesUserPassword
<project>-<env>-appsec/UnitsCacheRedisPassword
<project>-<env>-appsec/AzureEctokenKey
<project>-<env>-appsec/UnitsQueuesUserName
<project>-<env>-appcfg/DefaultFileStorage
<project>-<env>-taskcfg/DefaultFileStorage
<project>-<env>-taskcfg/DjangoMailerFromField
<project>-<env>-taskcfg/AWSAccountId
<project>-<env>-taskcfg/AppCacheRedisURL
<project>-<env>-taskcfg/DjangoMailerErrorFromField
<project>-<env>-taskcfg/UnitsCacheRedisURL
<project>-<env>-taskcfg/SiteDomain
<project>-<env>-taskcfg/SMTPMailServerHostname
<project>-<env>-taskcfg/SMTPMailServerPort
<project>-<env>-taskcfg/AWSRegion
<project>-<env>-taskcfg/SupportedProvisioningSoft
<project>-<env>-tasksec/UnitsQueuesUserPassword
<project>-<env>-tasksec/AppCacheRedisPassword
<project>-<env>-tasksec/EmailUseTls
<project>-<env>-tasksec/InitialValue
<project>-<env>-tasksec/AzureEctokenKey
<project>-<env>-tasksec/SMTPMailServerPassword
<project>-<env>-tasksec/UnitsQueuesUserName
<project>-<env>-tasksec/SMTPMailServerLogin
<project>-<env>-tasksec/UnitsCacheRedisPassword
<project>-<env>-tasksec/DjangoSecretKey
<project>-<env>-mhcfg/CloudUrl
<project>-<env>-mhcfg/RetryDelay
<project>-<env>-mhcfg/UnitRedisUrl
<project>-<env>-mhcfg/LogLevel
<project>-<env>-landing/InvitationUserKey
<project>-<env>-landing/RootCACertificate
<project>-<env>-landing/RecaptchaSecretKey
<project>-<env>-landing/AosCloudApiBaseURL
<project>-<env>-appsec/InvitationUserCertificate
<project>-<env>-landing/InvitationUserCertificates
<project>-<env>-base/MongoDbName
<project>-<env>-base/MongoAdmin
<project>-<env>-base/MongoAdminPassword
<project>-<env>-base/MongoUserPassword
<project>-<env>-base/MongoUser
<project>-<env>-data-services/MongoDbName
<project>-<env>-data-services/MongoReconnectionTimeLimit
<project>-<env>-data-services/InfluxDbPort
<project>-<env>-data-services/MongoHost
<project>-<env>-data-services/MongoPort
<project>-<env>-data-services/MongoUser
<project>-<env>-data-services/AlertsTtlDays
<project>-<env>-data-services/MongoUserPassword
<project>-<env>-data-services/InfluxDbHost
<project>-<env>-appcfg/ServiceUnitMonitoringUrl
<project>-<env>-appcfg/ServiceAlertUrl
<project>-<env>-data-services/InfluxDbToken
<project>-<env>-base/InfluxdbAdminToken
<project>-<env>-base/InfluxdbAdminPassword
<project>-<env>-data-services/InfluxDbOrg
<project>-<env>-data-services/InfluxDbBuket
<project>-<env>-base/AosRegistryProperties
<project>-<env>-base/RabbitMqSshKey
<project>-<env>-taskcfg/AwsEc2LaunchTemplate
<project>-<env>-taskcfg/ProjectNamespace
<project>-<env>-gitlabrunner/PrivateSshKey

AWS KMS Keys in AOS Edge

The following keys will be created and managed by terraform scripts.

• S3 Encryption -- AWS Managed KMS
• EC2 Volume encryption -- Alias: <project>-<env>-key
• EC2 Snapshots -- Alias: <project>-<env>-key
• Secrets Manager AOS related secrets -- Alias: <project>-<env>-key

AWS Secrets and Keys rotation

In case rotation of secrets it is needed you can do so by applying simple steps, terraform will take care of updating the credentials accordingly, some of the secrets can be taken from the env vars needed by terraform.

For secrets rotation:

1. Identify the secret to be rotated.
2. Change the secret in the terraform tfvars file or in the env var if you're loading them through env vars.
3. Rerun terraform scripts (terraform plan / terraform apply)
4. Restart deploy which depends of the secrets

   We recommend to restart all the pods in the <environment> variable namespace.

For keys KMS rotation:

1. terraform plan -replace module.kms_key.aws_kms_key.this
2. terraform apply -replace module.kms_key.aws_kms_key.this

Disclaimer WARN: KMS rotation will force new resource recreation, we don't recommend to rotate this key as it will be disruptive for more than one component in the AOS Edge, we had never tested this approach.

Data Encryption

All of the AWS Resources will be encryptied with the Customer KMS Key created as stated in the previous documentation section. This includes.

• S3 Objects are encrypted mandatory in Aos S3 Buckets Config.
• EC2 Snapshots created by velero are encrypted as well as the EKS Disks.
• In transit encryption it is forced for S3 communication.
• Encrypted communications are enforced from EKS pods to AWS services through istio.
• Istio cert it is self managed cert not provided by KMS.
• Secrets Manager encryption uses customer key described in previous section.

Public Resources

There are no publicly opened resources in the AOS Cloud for AWS implementation, all the S3 buckets should be private. The Cloudfront will generate signed url which is not public, and will be shared only to the authenticated devices.

Although the Network Load Balancer for the Istio Gateway it is public and can be accesses across internet, this will not allow the management access without proper Mutual TLS authentication, if this condition it is not met, it should not allow the access to the Aos Cloud web ui.

Network configuration

All the network resources and configuration can be found in the source code or packaged terraform code /terraform/infrastructure/network.tf.

These are the resources which will be configured by terraform automatically:

• NACL for base and aks subnet communications as well as service endpoints required.
  • Allow egress EKS
  • Deny Egress for base_vpc
• Security Groups:
  • Allow EKS to base
  • Allow EKS to internet egress
  • Allow MQ to EKS
  • Allow service endpoints to EKS
  • (Optional feature) Extra security group to attach custom rules for ingress in EKS/istio load balancer
• 2 VPC's: base and message queue
• Routes and route associations
  • EKS and endpoints
  • eks to mq
  • mq to base

IMDSv2

Pods: Aos Edge was developed with the latest EKS version which does not need IMDSv1 as requirement, meaning that Aos Cloud uses IMDSv2 as default option to get metadata from EKS pods, not instance itself, this approach ensures that we are using the latest IMDSv2 version.

RabbitMQInstances: By defect it uses the latest aws cli version which will be uninstalled from the instance and the instance will not have any role assigned, meaning that IMDSvX it is not needed at all by the EC2 instances provisioned by rabbit autoscaler.