Dynamic AWS Credentials with Vault

Dynamic AWS Credentials with Vault

August 18, 2022

Configuring AWS and Vault to utilize dynamic credentials is a multi-faceted process with multiple ways to consume the final outcome.

Base AWS resources are required that can then be consumed via Vault in order to feed back into AWS in the form of dynamic IAM users or assumed roles.

This post will outline my recent attempt to implement this for multiple AWS accounts and Vault clusters (though generic and distilled enough to relay the overall process).

The layout of this post follows the order resources were created in.

AWS #

IAM Users #

Each AWS account and Vault cluster has its own IAM user in order to keep access separated. This also allows for better control over which IAM users can assume which roles, further enforcing least-access.

Vault requires these IAM users have the ability to perform these IAM actions:

  "Action": [
    "iam:DeleteAccessKey",
    "iam:AttachUserPolicy",
    "iam:DeleteUserPolicy",
    "iam:DeleteUser",
    "iam:ListUserPolicies",
    "iam:CreateUser",
    "iam:CreateAccessKey",
    "iam:RemoveUserFromGroup",
    "iam:AddUserToGroup",
    "iam:ListGroupsForUser",
    "iam:PutUserPolicy",
    "iam:ListAttachedUserPolicies",
    "iam:GetUser",
    "iam:DetachUserPolicy",
    "iam:ListAccessKeys",
    "iam:ListRoles"
  ]

Given this, the policy is scoped to Vault-specific users:

"Resource": "arn:aws:iam::123456789012:user/vault-*"

Without this, the Vault IAM user could perform the above actions on any users, which would not be ideal.

The IAM users themselves are created with fairly basic resources and arguments:

resource "aws_iam_user" "vault_iam_root" {
  for_each = var.iam_user
  name     = format("vault-%s", each.key)
  tags     = var.tags
}

In this example, the key in the var.iam_user map would be example-user. The users are created with names to match with the Resource in the IAM policy JSON.

Access keys are then created for each user with a resource that also encrypts the secret access key with a PGP key:

resource "aws_iam_access_key" "vault_iam_root_access_key" {
  for_each = var.iam_user
  user     = aws_iam_user.vault_iam_root[each.key].name
  pgp_key  = var.encrypt ? var.pgp_key : null
}

When these credentials are created, they can be stored in Vault for later use which is recommended. After all, this whole effort revolves around Vault so it only makes sense to use it for this too.

IAM Policies #

The policies for each user are created with a straightforward resource that references local .json files rather than inline heredoc or an IAM Policy Document data resource:

resource "aws_iam_user_policy" "vault_iam_root_user_policy" {
  for_each = var.iam_user
  name     = format("vault-%s-policy", each.key)
  user     = aws_iam_user.vault_iam_root[each.key].name
  policy   = file("${path.module}/policy/${each.key}.json")
}

Each policy follows this general format (which includes the IAM permissions mentioned above while also allowing the vault-example-user IAM user to assume roles as necessary which is outlined further below):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "iam:DeleteAccessKey",
        "iam:AttachUserPolicy",
        "iam:DeleteUserPolicy",
        "iam:DeleteUser",
        "iam:ListUserPolicies",
        "iam:CreateUser",
        "iam:CreateAccessKey",
        "iam:RemoveUserFromGroup",
        "iam:AddUserToGroup",
        "iam:ListGroupsForUser",
        "iam:PutUserPolicy",
        "iam:ListAttachedUserPolicies",
        "iam:GetUser",
        "iam:DetachUserPolicy",
        "iam:ListAccessKeys",
        "iam:ListRoles"
      ],
      "Resource": "arn:aws:iam::123456789012:user/vault-*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "sts:AssumeRole"
      ],
      "Resource": [
        "arn:aws:iam::123456789012:role/vault-example"
      ]
    }
  ]
}

Roles #

In order to support assumed_role credential types in Vault, the Vault IAM user needs to be able to assume the necessary role(s) as well.

Once a role is created in the Console (or via Terraform/Cloudformation/etc.), the Vault user can be modified to allow access to the new role:

{
    "Sid": "VisualEditor1",
    "Effect": "Allow",
    "Action": [
      "sts:AssumeRole"
    ],
    "Resource": [
      "arn:aws:iam::123456789012:role/vault-example"
    ]
}

Take the vault-example role for instance – which can be created by an aws_iam_role resource:

resource "aws_iam_role" "vault-example" {
  count                = contains(tolist(keys(var.role)), "vault-example") ? 1 : 0
  name                 = "vault-example"
  max_session_duration = var.role["vault-example"].max_session_duration > 3600 ? var.role["vault-example"].max_session_duration : 3600
  assume_role_policy   = data.aws_iam_policy_document.vault_assume_role_policy.json
  managed_policy_arns  = [for k, v in data.aws_iam_policy.vault_example : v.arn]
}

This example is fairly contrived and can be made more dynamic if necessary.

Due to how the policy ARNs for the role are sourced and utilized in the role resource:

data "aws_iam_policy" "vault_example" {
  for_each = var.vault_example_policy
  name     = each.value
}

a clean way to support a for_each aws_iam_role resource is not feasible; therefore, each role needs to have its own data resource and its own aws_iam_role resource.

This has the potential to sprawl after a while, however.

The corresponding variables for the aws_iam_role and aws_iam_policy resources look something like this:

variable "vault_example_policy" {
  default = [
    "example-policy"
  ]
}

variable "role" {
  default = {
    ...
    vault-example = {
      max_session_duration = 43200
    }
    ...
  }
}

Vault #

With all of the AWS IAM resources in place, Vault can be configured. This involves an AWS Secrets Engine backend along with the backend roles for the engine.

AWS Secrets Engines #

In order to create dynamic credentials, Vault needs to be configured to access an AWS account. This is where the credentials created in the IAM Users section are used.

First, the credentials need to be sourced. The access key ID can be retrieved from the console or with the Terraform output; the secret access key can be decrypted with this command:

echo -n <encrypted output> | base64 -d | gpg -d | pbcopy

Once the credentials are retrieved, they can be placed in a KV path in Vault for use in the module that creates the necessary Vault resources for the AWS secrets backend.

The AWS secret backend resource looks like this:

data "vault_generic_secret" "vault_root_iam" {
  for_each = var.aws_secret_backend
  path     = each.value.vault_path
}

resource "vault_aws_secret_backend" "aws" {
  for_each                  = var.aws_secret_backend
  access_key                = data.vault_generic_secret.vault_root_iam[each.key].data["access_key"]
  secret_key                = data.vault_generic_secret.vault_root_iam[each.key].data["secret_key"]
  region                    = each.value.region
  path                      = each.value.path
  description               = each.value.description
  default_lease_ttl_seconds = each.value.ttl
  max_lease_ttl_seconds     = each.value.max_ttl
  iam_endpoint              = each.value.iam_endpoint
  sts_endpoint              = each.value.sts_endpoint
}

The IAM user credentials are sourced from the data resource and used in the vault_aws_secret_backend resource.

The input for the AWS secret backend looks like this:

backend = {
  region            = "us-east-1"
  vault_path        = "path/to/iam/root/credentials"
  path              = "aws/account"
  description       = "The AWS Secrets Engine backend for dynamic users in AWS Account 123456789012"
  ttl               = 3888000
  max_ttl           = 7776000
  iam_endpoint      = null
  sts_endpoint      = null
  username_template = <<EOT
{{ if (eq .Type "STS") }}
    {{ printf "vault-%s-%s" (unix_time) (random 20) | truncate 32 }}
{{ else }}
    {{ printf "vault-%s-%s-%s" (printf "%s-%s" (.DisplayName) (.PolicyName) | truncate 42) (unix_time) (random 20) | truncate 64 }}
{{ end }}
EOT
  }
}

This input sets the KV path for the IAM user credentials, the default TTL, the max TTL, and how the dynamic usernames are formatted. Each AWS account that needs to be added requires its own backend.

Backend Roles #

Once the AWS secret backends are configured, backend roles can be created in their respective backends. These can either be of type iam_user or assumed_role as mentioned previously.

A backend can have many roles.

By default the module will create backend roles as iam_user types since this is the most common (and simplifies how much configuration is required when creating the code for each backend role).

The resource that creates the each backend role looks like this:

resource "vault_aws_secret_backend_role" "aws_role" {
  depends_on = [
    vault_aws_secret_backend.aws
  ]
  for_each        = var.aws_secret_backend_role
  backend         = each.value.backend != null ? each.value.backend : "aws/account"
  name            = each.value.name
  credential_type = each.value.credential_type != null ? each.value.credential_type : "iam_user"
  iam_groups      = each.value.iam_groups != [] ? each.value.iam_groups : []
  role_arns       = each.value.credential_type == "assumed_role" || each.value.role_arns != null ? each.value.role_arns : []
  policy_arns     = each.value.credential_type == "assumed_role" || each.value.credential_type == "federation_token" || each.value.policy_arns != null ? each.value.policy_arns : []
  policy_document = each.value.policy_document != "" ? each.value.policy_document : ""
  default_sts_ttl = each.value.credential_type == "assumed_role" || each.value.credential_type == "federation_token" ? each.value.sts_ttl : null
  max_sts_ttl     = each.value.credential_type == "assumed_role" || each.value.credential_type == "federation_token" ? each.value.max_sts_ttl : null
}

iam_user backend role inputs are defined like this in the module-caller:

readonly = {
  name = "readonly"
  policy_arns = [
    "arn:aws:iam::aws:policy/ReadOnlyAccess"
  ]
}

assumed_role backend role inputs are defined like this:

role = {
  backend         = "aws/account"
  name            = "vault-role"
  credential_type = "assumed_role"
  role_arns = [
    "arn:aws:iam::123456789012:role/vault-role"
  ]
  sts_ttl     = 3600
  max_sts_ttl = 43200
}

For assumed_role credential types, sts_ttl and max_sts_ttl are required.

While not used in the iam_user example, an IAM Group can be specified in addition to the policy ARNs if necessary.

Multiple policy ARNs can also be specified.

In order to grant access to specific backend roles, policies are also created in a 1:1 fashion so that Vault entities can be granted access to a single role.

A resource that creates these policies may look something like this:

resource "vault_policy" "aws_secret_backend_role_policy" {
  for_each = vault_aws_secret_backend_role.aws_role
  name     = format("%s.%s", replace(each.value.backend, "/", "-"), each.value.name)
  policy   = <<-EOT
  ### AWS Secret Backend Role Policies ###
  # Allow listing of a specific AWS secret backend role
  path "${each.value.backend}/roles/${each.value.name}" {
    capabilities = ["list"]
  }
  # Allows access to a specific AWS secret backend role
  path "${each.value.backend}/creds/${each.value.name}" {
    capabilities = ["read"]
  }
  EOT
}

Bucket Policy Example #

Accessing S3 buckets with dynamic iam_user credentials is made difficult by the ephemeral nature of dynamic IAM users which is difficult to capture within a bucket policy. To avoid this entirely, assumed_roles (e.g., vault-example) can be used.

An example bucket policy (using the aws_iam_policy_document data resource) may look something like this:

data "aws_iam_policy_document" "policy" {
  statement {
    sid    = "DenyAllExceptListed"
    effect = "Deny"

    resources = [
      "arn:aws:s3:::bucket-name",
      "arn:aws:s3:::bucket-name/*",
    ]

    actions = [
      "s3:ListBucket",
      "s3:Abort*",
      "s3:Bypass*",
      "s3:Delete*",
      "s3:Put*",
      "s3:Replicate*",
      "s3:GetObject",
    ]

    condition {
      test     = "StringNotLike"
      variable = "aws:userId"

      values = [
        "123456789012",            // root account
        "AIDA...",                 // traditional IAM user IDs
        "AROA...:*"                // role IDs and assumed roles
      ]
    }

    principals {
      type        = "*"
      identifiers = ["*"]
    }
  }
}

resource "aws_s3_bucket_policy" "s3_bucket_policy" {
  bucket = aws_s3_bucket.s3_bucket.id
  policy = data.aws_iam_policy_document.policy.json
}

The most important area of the IAM Policy Document is the condition:

condition {
  test     = "StringNotLike"
  variable = "aws:userId"

  values = [
    "123456789012",            // root account
    "AIDA...",                 // traditional IAM user IDs
    "AROA...:*"                // role IDs and assumed roles
  ]
}

which explicitly allows for role IDs and wildcards to be used in the policy.

GitHub Actions Example #

In order to support dynamic credentials in GitHub Actions Workflows, something like the following can be added:

- name: Retrieve dynamic credentials
    id: dynamic_creds
    run: |
      AWS_CREDS=$(VAULT_ADDR="${{ inputs.vault_cluster }}" ./vault read aws/${{ inputs.aws_backend }}/creds/${{ inputs.backend_role }} -format=json)
      echo "AWS_ACCESS_KEY_ID=$(echo $AWS_CREDS | jq '.data | .access_key' | tr -d "\"")" >> $GITHUB_ENV
      echo "AWS_SECRET_ACCESS_KEY=$(echo $AWS_CREDS | jq '.data | .secret_key' | tr -d "\"")" >> $GITHUB_ENV
      echo "AWS_SESSION_TOKEN=$(echo $AWS_CREDS | jq '.data | .security_token' | tr -d "\"")" >> $GITHUB_ENV
      echo "LEASE_ID=$(echo $AWS_CREDS | jq '.lease_id' | tr -d "\"")" >> $GITHUB_ENV

These environment variables can then be used in a Configure AWS Credentials step:

- name: Configure AWS Credentials
    uses: aws-actions/configure-aws-credentials@13d241b293754004c80624b5567555c4a39ffbe3
    with:
      aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }}
      aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }}
      aws-session-token: ${{ env.AWS_SESSION_TOKEN }}
      aws-region: ${{ inputs.region }}

This has the unfortunate side effect of rendering the secrets in plaintext, but due to their short-lived nature, this should not present a huge issue because the lease can be revoked with the following step (even if previous steps fail):

- name: Revoke AWS Credential Lease
    run: VAULT_ADDR="${{ inputs.vault_cluster }}" ./vault lease revoke ${{ env.LEASE_ID }}
    if: always()

Diagrams #

To tie everything in together, this diagram shows how resources feed into and reference each other:

flowchart TD IP[Vault User IAM Policy] --> |Attached To| IU[Vault IAM User] EIP[Existing IAM Policies] --> |Attached To| EIR[Existing IAM Role] EIR[Existing IAM Role] --> |Assumed By| IU[Vault IAM User] IU[Vault IAM User] --> |Used By| ASB[AWS Secret Backend] ASBR[AWS Secret Backend Role] --> |Created In| ASB[AWS Secret Backend] ASBR[AWS Secret Backend Role] --> |Attaches| EIP[Existing IAM Policies] VE[Vault Entity] --> |Granted Access To| VP[Vault Policy] --> |Grants Access To| ASBR[AWS Secret Backend Role] GHW[GitHub Workflow] --> |Reads From| ASB[AWS Secret Backend] ASB[AWS Secret Backend] --> |Creates| DIU[Dynamic IAM User/Session] --> |Can Access| AR[AWS Resources]