Adventures in Terraform State

May 1, 2022

Maintaining [older] Terraform versions and custom modules can sometimes spiral into scenarios of various complexities.

For instance, a recently outdated module with the following module-caller:

module "original_reference" {
	source = "git::[email protected]:org/tf-module-repo.git//path/to/module?ref=x.y.z"
	input1 = var.input1
	input2 = var.input2
	...
}

can become problematic if the underlying module reference changes (and is still on Terraform 0.12.x – more on that later).

In this context, the original module and the new module were technically the same with the only difference being the name. The above code was deleted in favor of using a module that looked like this (along with a newer source ref):

module "new_reference" {
	source = "git::[email protected]:org/tf-module-repo.git//path/to/module?ref=x.y.z"
	input1 = var.input1
	input2 = var.input2
	...
}

This issue reared its head the other day when the module-caller had its underlying module reference updated (using module.new_reference instead of module.original_reference) on top of being upgraded from 0.12.x to Terraform 1.0.x. The Terraform version being outdated is not directly related to the module reference issue, but happened to coincide with this.

This resulted in this fairly obscure error which is helpfully documented in Terraform’s v0.13 upgrade documentation:

Error: Provider configuration not present

To work with <resource> its original provider configuration at
provider["registry.terraform.io/-/aws"] is required, but it has been removed.
This occurs when a provider configuration is removed while objects created by
that provider still exist in the state. Re-add the provider configuration to
destroy aws_instance.example, after which you can remove the provider
configuration again.

The above error occurs when initially running terraform init.

But how did the provider config get removed?

Turns out, the module changing names resulted in Terraform losing track of the module in state; Terraform expected module.original_reference rather than module.new_reference. Additionally, the Terraform 0.13upgrade steps were not performed, but addressing both is relatively simple.

To address the as-of-now hidden 0.13.x upgrade requirements at this stage, the following command replaces the provider to the new pattern for Terraform versions >= 0.13:

❯ terraform state replace-provider 'registry.terraform.io/-/aws' 'registry.terraform.io/hashicorp/aws'

This will (read: should) allow the Terraform version to be upgraded directly from 0.12.x to whatever version of Terraform without requiring an incremental upgrade to 0.13.x (or at least running terraform 0.13upgrade) and then to the desired version. This was performed with a single provider; multiple providers may complicate or break this workaround.

As for the module changing from module.original_reference to module.new_reference, a quick terraform state mv can rectify the drift:

❯ terraform state list

# make note of module resources

❯ terraform state mv 'module.original_reference.resource.name' 'module.new_reference.resource.name'
❯ terraform state mv 'module.original_reference.list_resource.name[0]' 'module.new_reference.list_resource.name[0]'
❯ terraform state mv 'module.original_reference.list_resource.name[1]' 'module.new_reference.list_resource.name[1]'
❯ ...

After which, terraform plan should run cleanly and display any original changes that were trying to be made.

The resources at the new path(s) can be verified by running:

❯ terraform state show 'module.new_reference.resource.name'

Generally, Terraform state should not be modified (or need to be modified), but doing so can prove extremely useful in eliminating headaches when something happens behind the scenes. If a state mv does not work, it should be possible to move the resource back to its original path in state.

As a bonus, doing this in an environment or situation where the backend.tf file looks like this:

terraform {
  backend "s3" {}
}

can make resolving the issue a little more difficult.

In such cases, Terraform State can be initialized using -backend-config arguments:

❯ terraform init -backend-config="region=<>" \
	-backend-config="bucket=<>" \
	-backend-config="key=<>" \
	-backend-config="dynamodb_table=<>" \
	-backend-config="encrypt=true"

This example assumes an S3 backend and DynamoDB lock table with encryption are being used.