How Terraform 0.13 will change my code

9 minutes read
HashiCorp • Terraform • Infrastructure as Code

HashiCorp has recently released the first beta of Terraform 0.13, exactly a year after the major overhaul release of 0.12. Although smaller than 0.12, this new version of Terraform is introducing many changes I have been looking for for years, and I am going to give an overview of how they will change the way I write code and interract with the tool.

Headline features

For_each expressions for modules

Terraform 0.12 improved on existing count concept, which is used to create multiple copies of a resource easily, and introduced for_each. Just like count however, for_each continued to only work for resources and data sources, Terraform 0.13 is introducing it for modules as well.

I use Terraform to create and manage my code repositories, and I use a custom module for creating and configuring a repository to my liking. With Terraform 0.12, the code to create all of the repositories looks like this:

module "terraform-aws-cool-module" {
    source  = "example.com/terraform-gitlab-repository"
    version = "1.0.0"

    name       = "terraform-aws-cool-module"
    visibility = "public"
}

module "secret-workspace" {
    source  = "example.com/terraform-gitlab-repository"
    version = "1.0.0"

    name       = "secret-workspace"
    visibility = "private"
}

The code above works very well. I can add a new repository by just adding another module block to the file, and changing the values to match what I want. As soon as I want to add a new parameter to every repository, or update the module version, I have to edit dozens of identical blocks of code though.

Support for for_each parameter at module level allows me to simplify the code to this:

locals {
    repositories = {
        "terraform-aws-cool-module" = {
            public = true
        }
        "secret-workspace" = {}
    }
}

module "repository" {
    for_each = local.repositories

    source  = "example.com/terraform-gitlab-repository"
    version = "1.0.0"

    name       = each.key
    visibility = lookup(each.value, "public", false) ? "public" : "private"
}

Even though both solutions work well for my use case, for_each support for modules allow workflows that were very cumbersome before, because it allows you to combine modules and dynamic data sources. Some of the examples where this feature would have been especially useful to have for me:

  • Retrieve a list of account numbers in an AWS Organizations OU with a data source, and invite those accounts to GuardDuty;
  • Retrieve a list of subnets in a VPC with a data source, and deploy a service to each subnet.

Depends_on with modules

Some resources in Terraform have to be constructed in certain order, when creating a DNS Zone and a DNS record in the same run, Terraform must create the Zone before the record, because one depends on another. Normally, these dependencies are discovered from variable interpolation, and Terraform creates the resources in correct order automatically, but some resources have hidden dependencies on others, and need to be declared explicitly using depends_on parameter:

One example of such hidden dependency is Config service in AWS. Before any AWS Config rules can be enabled, the service needs to be enabled, done by config-recorder module:

module "config-recorder" {
    source  = "example.com/config-recorder"
    version = "1.0.0"
}

module "config-rules" {
    source  = "example.com/config-rules"
    version = "1.0.0"
}

Before depends_on support, the way to create a dependency between the modules was to use dummy interpolation values and null_resource to trigger changes in config-rules module, but with Terraform 0.13, this becomes as simple as:

module "config-rules" {
    source  = "example.com/config-rules"
    version = "1.0.0"

    depends_on = [
        module.config-recorder,
    ]
}

Although I do not expect to regularly use depends_on with modules, as try to use variable interpolation to define dependencies as much as possible, but when interpolation is unavailable, depends_on is a life saver.

Additional module sources

Terraform 0.13 gets support for retrieving modules for non-official providers.

Up until now, if you wanted to use a custom provider, or use terraform in air-gapped environment, you had to download the provider separately and place it either in $PATH, or in terraform plugins directory. This is especially difficult in Terraform Cloud or Terraform Enterprise, where to use a custom provider you need to either commit the provider in code, or build a custom terraform package with terraform bundle.

I have not been able to test this functionality yet, but once Terraform module registry, and especially Terraform Private Module Registry, allows publishing providers, I can see myself immediately starting to use them to distribute my custom providers.

Variable validation

Introduced in Terraform 0.12 as an opt-in experiment, variable value validation is generally available in Terraform 0.13.

Variable validation allows creators of modules to check whether values have correct format at plan time, and provide informative error messages to the users.

I have a reusable module to deploy a service, which creates a Target Group in AWS with a custom name. Target Groups in AWS have a very low character limit of 32 characters, and because of additional information the module adds to the name, the user configurable part is only 10 characters in length. The code in the module looks like this:

resource "aws_lb_target_group" "lambda-example" {
  name        = "${local.environment}-${var.stack_name}-${local.random_hex}"
  ...
}

If I try to supply a name longer than 10 characters, the code above already catches the error at plan time because of validation done by the provider: Error: "name" cannot be longer than 32 characters. Although this correctly catches the problem, the error message requires you to understand the internals of the module to find out what can you do to fix it. Variable validation fixes that:

variable "stack_name" {
  type = string
  validation {
    condition     = length(var.stack_name) <= 10
    error_message = "Stack name should not be longer than 10 characters."
  }
}

Running the plan again gives me a more descriptive error message, not only describing the exact limit, but also which variable in my code violates it:

Error: Invalid value for variable

  on main.tf line 43, in module "tile-ingress-service":
  43:   name = "tile-ingress"

Stack name should not be longer than 10 characters.

This was checked by the validation rule at service/variables.tf:8,3-13.

Although I do not expect to have to use this feature a lot for my own projects, terraform providers perform enough validation already, and I can fit all of my own source code in my head, I can see it being especially useful for module authors, and people working in large teams.

Minor changes

As well as the major headline features, which will change the way many people interract with Terraform, there are several smaller changes that I have noticed that will affect me too.

Destroy provisioner references

I sometimes find myself needing to perform some custom actions in the middle of a terraform run, but find writing a provider for a resource or two just to have too much overhead. In such cases, I resort to null_resource and local-exec provisioners to implement it directly in terraform configuration:

resource "null_resource" "resource" {
    triggers {
        resource_name = local.name
    }

    provisioner "local-exec" {
        command = <<EOC
            python create-resource.py "${var.api_url}" "${var.api_key}" "${local.name}" "${local.data}"
EOC
    }

    provisioner "local-exec" {
        when    = destroy
        command = <<EOC
            python delete-resource.py "${var.api_url}" "${var.api_key}" "${local.name}"
EOC
    }
}

The code above allows wrapping a target API with a few python scripts, and incorporate them into your Terraform workflow. Although not as full featured as a provider (does not support state refresh for one), it is usually sufficient. For a more full featured implementation, see terraform-shell-resource module.

Starting with Terraform 0.12 the above configuration produced the following warning, and in Terraform 0.13 it has been promoted to an error:

Destroy-time provisioners and their connection configurations may only
reference attributes of the related resource, via 'self', 'count.index', or
'each.key'.

Requiring destroy provisioners to only depend on the resource itself is a welcome improvement in Terraform, as it helps prevent very confusing dependency cycles, even if it makes abusing provisioners for our purposes harder.

Fixing Destroy provisioner

If you have been using destroy provisioners and null_resource for this purpose, here are the ways to fix your code for Terraform 0.13:

Embed all variables used in the provisioners in the triggers parameter. You are then able to access them via self.triggers in the destroy provisioner:

resource "null_resource" "resource" {
    triggers {
        api_url = var.api_url
        api_key = var.api_key
        name    = local.name
        data    = local.data
    }

    provisioner "local-exec" {
        command = <<EOC
            python create-resource.py "${self.triggers.api_url}" "${self.triggers.api_key}" "${self.triggers.name}" "${self.triggers.data}"
EOC
    }

    provisioner "local-exec" {
        when    = destroy
        command = <<EOC
            python delete-resource.py "${self.triggers.api_url}" "${self.triggers.api_key}" "${self.triggers.name}"
EOC
    }
}

The change above will allow you to continue using the same code as before with no code change and minimal configuration change. If anything, the destroy provisioner will work more consistently. A downside of this approach is that any sensitive values, such as api_key in the example, are now stored in the state in plain text and can be easily extracted by people with access to the state. Before using this solution, read the note about Sensitive Data in State in Terraform documentation.

An alternative to this is to write your own custom provider to perform the actions you want. Although it involves more coding, it provides complete control over what to store in the state, allows implementing resource state refresh and import, and the full power of Terraform API. With the improvements to provider sources in Terraform 0.13, this has been made even easier than ever, start at Writing Custom Providers in Terraform documentation.

Terraform Cloud improvements

When using a remote backen, used by Terraform Cloud and Terraform Enterprise, Terraform 0.13 supports two new commands:

  • terraform state push -force;
  • terraform plan -target <resource>.

Both of these commands have been available for other Terraform backends already, but were not possible to replicate without tricks when using Terraform Cloud.

Force state push

Every once in a while I find myself with a corrupted state file in Terraform Enterprise, which requires me to restore the state to the previous valid version. Unfortunately, since the state is corrupted, all ways to push old state to the workspace fail with differing lineage errors, and require doing either:

  • Deleting and recreating a workspace from scratch, which loses the history and secret variables, or
  • Using undocumented Force parameter in Terraform Cloud API to override the check.

The terraform state push -force command brings feature parity with other backends to Terraform Cloud, and makes recovering state significantly easier. I will surely use this command, instead of the alternatives, the next time I encounter a state error.

Targeted plans

Multiple times I have been in a situation where a change outside of Terraform makes a workspace unable to apply cleanly, and I have to publish an unrelated change that is contained in the same workspace.

In other Terraform backends, and as of this change, in Terraform Cloud/Enterprise, the failing resource can be temporarily ignored, and the other resources can be applied using targeted plan:

terraform plan -target <resource-1> -target <resource-2> ...

The resulting plan will only include targeted resources, and any of their dependencies. As long as the failing resource is not related to them, this allows you to continue the change, and leave fixing the failing part for later.

As of writing this blog post, the targeted plan feature for Terraform Enterprise has not been released, but I cannot wait to use it once it becomes available.