Password-Handling in AWS and IaC

What to do if you need good-old password management for AWS? Introducing AWS secrets manager.

One of the common hassles of every developer is how to handle passwords and other secrets appropriately – This means maintainable but first and foremost secure. When you are in AWS, you want to use the built-in permission system of IAM with roles and permission delegation as much as possible – But sometimes you cannot get around good-old password management.

AWS provides a secret store for this called AWS secrets manager. A similar offering exists for other cloud vendors. So, those shall be used in the cloud, and like every other resource, it shall not be configured manually but as Infrastructure-as-Code (IaC), right? Before we look at some examples, we need to talk about need-to-know.

Need to know

A basic principle in the world of security is called need-to-know. It means you should only grant access to a secret to the resources that need to work with it to keep the information as contained as possible. A resource here includes software as well as humans. If two systems interact securely using a password, why should you as a human know about it? It might feel safer to know all the passwords, but it actually increases the security exposure.

If you think you might need access for debugging or an emergency situation, that is OK – You just need to rotate the password afterward without knowing the new value.

Let’s take a look at a couple of examples, and see how we can establish IaC and password rotation.

Example #1 – RDS – The easy way

One of the major use-cases for good-old passwords is to connect to databases. Fortunately, AWS provides a very handy solution. When creating a database, AWS can automatically create and manage a secret in AWS secrets manager for you. This means:

The secret resides in AWS secrets store and can be retrieved by your application from there (using IAM) The secret can be automatically rotated, by default every seven days. Your application needs to support it, but that is easy. If you try to connect and get a ‘bad password’ error, simply re-read the secret. It probably has rotated. Goodbye long-lived credentials. You as a human don’t ever need to know the secret, and you can even set up your AWS permissions so that you cannot read it (unless you have admin privileges in the account). Is this procedure violating the goal of infrastructure-as-code? I don’t think so. IaC means you can manage your whole infrastructure via code, not that every resource needs to be explicitly defined. Let’s say you use Terraform and you declare your RDS instance like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

resource "aws_db_instance" "example" {
  allocated_storage           = 10
  db_name                     = "mydb"
  engine                      = "mysql"
  engine_version              = "5.7"
  instance_class              = "db.t3.micro"
  manage_master_user_password = true
  username                    = "foo"
}

output "db_secret_arn" {
  value = aws_db_instance.example.master_user_secret[0].secret_arn
}

Terraform will create the RDS instance with a master user called ‘foo’. The corresponding password will be stored in the secrets manager as a distinct secret. The password will be encrypted with the default KMS key, as we did not define a custom one. We can now reference the password with the AWS ARN, but the actual password value is not exposed ever. We can use this ARN in other infrastructure areas via IaC.

The secret entry will be automatically deleted when the RDS instance gets deleted. No manual interaction with the secret is needed throughout the whole RDS lifecycle – That is IaC.

Example #2 – Apache ActiveMQ – We need a workaround

Let’s now take a look at ActiveMQ, a popular open source message broker. When setting up, you need to define a username/password for further configuration and access. Unfortunately, even in the AWS-managed version, you need to manually manage the secret.

What you should NOT do here is use Terraform like this :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

resource "random_password" "password" {
  length = 16
  special = true
  override_special = "_%@"
}


resource "aws_mq_broker" "example" {
  broker_name = "example"

  engine_type        = "ActiveMq"
  engine_version     = "5.17.3"
  storage_type       = "ebs"
  host_instance_type = "mq.m5.large"

  user {
    username = "admin"
    password = random_password.password.result
  }
}

Why? In order to detect differences between the desired and actual state, Terraform utilizes an intermediate ‘state file’. In this file, the password is stored in clear text like this (somewhere in the config block for the aws_mq_broker resource):

1
2
3
4
5
6
7
8
9

"user": [
              {
                "console_access": false,
                "groups": [],
                "password": "Lig1_kWpU9rSRV_U",
                "replication_user": false,
                "username": "admin"
              }
            ]

You can encrypt and restrict access to the state file, but this creates extra effort and risk. You could also use the commercial version of Terraform, where they manage the state file for you. I personally believe it’s always better to leave it secret-free. If you do want to store secrets in the state, please at least ensure they are defined as sensitive.

BTW: Some people are complaining that secrets are stored in the state file, but I think they are missing the whole point of Terraform. Terraform needs to store all the relevant information somewhere to be able to manage the infrastructure, for example to detect changes. This is independent of how the configuration took place – hard-coded strings, resource reference or Terraform variables. They all show up in the Terraform state. So, what we want to do is manage the resources in Terraform, but the actual secret value outside of Terraform. Professional/commercial solutions like Vault exist, but that might be too complicated for simple use-cases. Let’s keep it simple and do this instead:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

resource "aws_secretsmanager_secret" "mq_secret" {
  name = "mq_secret"
}

resource "aws_mq_broker" "example" {
  broker_name = "example"

  engine_type        = "ActiveMQ"
  engine_version     = "5.17.3"
  storage_type       = "ebs"
  host_instance_type = "mq.m5.large"

  user {
    username = "admin"
    password = "IGNORE-ME-I-AM-ALREADY-OUTDATED."
  }
  lifecycle {
  ignore_changes = [
      user,
    ]
  }
  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    command     = "./update_secrets.sh ${aws_secretsmanager_secret.mq_secret.name} ${aws_mq_broker.example.id}"
  }
}

In addition to before, we create an empty AWS secret and create a dummy value for our MQ password. The ignore_changes keyword means Terraform will not react to changes to the user configuration, including the password. That means the original, dummy password will stay in the state forever. So far, so good. But how do we set a meaningful and secret password? This is where local-exec comes into play. It tells Terraform to run an external script once the resource is created, and only then. Our script looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#!/bin/bash
set -eu

if [ $# -ne 2 ]; then
    echo "Script to update a password both in secretsmanager and "
    echo "Active MQ managed for our admin user."
    echo "Usage: $0 <aws_secretsmanager_secret_id> <broker_id>"
    exit 1
fi

AWS_SECRETSMANAGER_ID=$1
MQ_BROKER_ID=$2

echo "Generating a random password string using AWS secrets manager"
echo "--- Length 12, no special chars ---"
NEW_PASSWORD=$(aws secretsmanager get-random-password --exclude-punctuation \
    --password-length 12 --query 'RandomPassword' --output text)
echo "Updating secret in AWS secrets manager ${AWS_SECRETSMANAGER_ID}"
aws secretsmanager put-secret-value --secret-id  ${AWS_SECRETSMANAGER_ID} \
    --secret-string "${NEW_PASSWORD}" > /dev/null
echo "Done, updating secret in ActiveMQ ${MQ_BROKER_ID}"
aws mq update-user --broker-id ${MQ_BROKER_ID}  --username "admin" \
    --password ${NEW_PASSWORD}

It’s actually only doing three things:

  • Generate a new password via the AWS CLI
  • Store that password in secrets manager
  • Set the password as the new ActiveMQ password

When this script is executed successfully, the password will be updated in both places, so our application accessing ActiveMQ can get it from there. You can also rotate the password when you run the script again.

Example #3 – Using a password from outside – Ugh.

Sometimes you have to use a password from an external source, for example to connect to a 3rd party system. You don’t have much of a choice here. You can still store in secrets manager, and store the password using aws secretsmanager put-secret-value, similar to the example above. This also means the password is actually stored in two places (Your secure storage in the CICD system and secrets manager), but you can set your environment up automatically.

Obviously, you cannot rotate the password, as you don’t own it.

If you have a good vendor relationship, you might ask them to do the following:

  • Set up an AWS account
  • Store the secret in this account
  • Allow access to it from your account via cross-account permissions
  • Agree on password rotation

Final notes

If you remember one single thing out of this article, I hope it is the following : Your handling of passwords and other secrets needs to be based on the actual need-to-know, and you likely need to implement different procedures. Don’t try for a one-size-fits-all solution if the need-to-know is different!

Über den Autor

img/team/lohausen.jpg Daniel Lohausen

Daniel Lohausen

Cloud-Architekt und Prozess-Coach

Als Cloud-Architekt und Prozess-Coach vereint Daniel Lohausen zwei Kernelemente erfolgreicher Softwareentwicklung in einem ganzheitlichen Beratungsansatz. Er ist überzeugt, dass sich technische und prozedurale Innovationen positiv beeinflussen und man die besten Ergebnisse erzielt, wenn man bei beide Aspekte konstant weiter entwickelt.

Alle Artikel