Lock module versions, like providers, in .terraform.lock.hcl

[jcarlson]

Current Terraform Version

>= 0.14

Use-cases

In order to ensure that the versions of modules selected by terraform init is repeatable across multiple machines and multiple environments, which may run apply at different times, it would be useful to include module versions in the lock file.

Attempted Solutions

An obvious solution would be to specify exact versions for modules and update as necessary. However, we have switched hosting our in-house modules to a private Terraform Registry in order to take advantage of semantic versioning requirements, such as ~> 1.2 for a module.

We do have a few nested modules where for example the root depends on module A @ 1.2.0, which depends on module B @ 2.4.0. If module B needs to have a security fix released in 2.4.1, we would then have to issue a release to module A @ 1.2.1, and finally update the root configuration, which is time consuming and not ideal.

Proposal

Dependency managers for most other programming languages such as Ruby and Node already handle this sort of transitive dependency management, so it makes sense for Terraform's dependency management to do this as well.

The specific version of dependent and transient modules resolved by terraform init should be recorded to a lock file included in version control in order to improve repeatability across machines, environments and time.

Module versions can be upgraded with terraform init -upgrade similar to how providers can be upgraded.

References

• Pin modules by Commit SHA #29867

[apparentlymart]

Thanks for sharing this use-case, @jcarlson.

I was initially tempted to merge this into #31134, because the challenges I described in a comment there are also why we didn't end up managing to include module dependencies in the lock file as we'd originally hoped, but on reflection it seems like the two use-cases are separate even though the underlying problems are not and so I'll just create a link between them with this comment instead.

Using exact version constraints is indeed our recommended approach for now, but I think we would like to find a way to introduce more centralized management of module dependencies at some point, just that we need to find a design that is in some sense backward compatible with existing configurations due to the Terraform v1.0 Compatibility Promises.

Thanks again!

[jcarlson]

It feels like all the contracts are in place today to accommodate this. The existing .terraform.lock.hcl file seems like an ideal place to include metadata about locked modules. Something like:

provider "registry.terraform.io/hashicorp/aws" {
  version     = "3.74.3"
  constraints = ">= 3.0.0, < 4.0.0"
  hashes = [
    "a hash of the provider binary",
  ]
}

module "my.module.address" {
  version     = "1.2.0"
  constraints = "~> 1.2.0"
  hashes = [
    "a hash of the module source code"
  ]
}

I'm not at all familiar with the internal working of Terraform, so pardon any ignorance here, but assuming such a feature were added to a future version of Terraform, say 1.4 or 1.5, this doesn't feel like it would be cause for significant backwards-compatibility problems. Older versions of Terraform would simply not respect locked versions of modules, much the same way Terraform 0.13 does not lock providers, but 0.14 does.

[jcarlson]

What I mean to say is: in what way does adding module locking to Terraform 1.x violate any compatibility promise? Does the compatibility promise mean that a configuration written for Terraform 1.8 should work with Terraform 1.2?

[apparentlymart]

Hi @jcarlson,

In the past requests similar to this one have been framed as wanting to lock modules on a per-source basis, rather than on a per-call basis. In other words, the label on the module block in the file would be something like registry.terraform.io/hashicorp/consul/aws, rather than my.module.address as in your example. And upgrading to a newer version of registry.terraform.io/hashicorp/consul/aws would then presumably simultaneously update every module call that has source = "registry.terraform.io/hashicorp/consul/aws" at once.

I'm sorry for reading a requirement into your use-case that wasn't actually there. However, given that we've heard various different desires around this over the years I think whatever we do the first step will be to research and find out what the broad needs are. We're currently in some sense "spoiled" by each call being totally independent of the others and so we have the ultimate flexibility to upgrade each call separately, but what some see as a benefit others see as the burden of having to update many separate configurations each time they want to adopt a new version of a module. A solution here should, I think, ideally aim to help those with both perspectives, or at least enable building tools that could work alongside Terraform for coordinating upgrades (such as dependabot).

While I would agree that what you described would not be a breaking change on its face, it also isn't necessarily a design that meets all of the requirements. I cannot say for sure because I don't have all of the requirements to hand, but a design effort in this area will hopefully collect some other perspectives and help us figure out what the best compromise is.

(When we introduced the provider dependency lock file in the first place we did have some folks who reacted as if we had made a breaking change, because they had got accustomed to a workflow where providers were not locked. We do need to be cognizant of that sort of reaction here too, but I expect that as with the dependency lock file that could come in the form of documenting a way that people can bypass the locking of modules if they wish, rather than being constrained to not do it at all. We'll see!)

[jcarlson]

Thanks @apparentlymart. Yes just to add clarity for my use case, I would expect that a single root configuration could call multiple instances of a module and specify different version constraints for each independently. e.g.:

module "foo" {
  source  = "app.registry.io/my-org/my-module/aws"
  version = "~> 1.2"
}

module "bar" {
  source  = "app.registry.io/my-org/my-module/aws"
  version = "~> 2.0"

  feature_opt_in = true
}

In this example, modules foo and bar both use the same source address, but module bar requires a newer version in order to take advantage of some new features available in 2.0.0. Meanwhile, foo does not require this functionality and thus does not need to upgrade.

Perhaps in this example, module foo was created some time ago, but upgrading to v2.0.0 would be a breaking change involving the destruction and recreation of critical resources. Module bar was added later, but since it is greenfield, it need not be limited to the older version constraints that foo has.

[redzioch]

I solved this using separated git repositories for versioned modules. Module releases are tagged, so I can stick to different versions of one module.

It looks like this:

module "test" {
  source = "git::ssh://git@gitlab/sample-terraform-module.git?ref=1.0.0"
  (...)

[jcarlson]

@redzioch unfortunately, using Git tags as a module source does not allow for semantic version ranges to be expressed. For example, you can’t set a version constraint of ~> 1.3 using Git tags.

This also requires that for nested modules, you must update each intermediate module and create a release. This is not ideal for a root configuration that depends on module A, which in turn depends on module B. Updating module B would require touching code in modules B, A and the root.

While you could maintain Git tags corresponding to v1.3.0, v1.3 and v1, and update the latter two over time, this doesn’t achieve the goal here which is to include a lock file for modules, so that two separate machines running terraform init at separate times chronologically are guaranteed to resolve the same version of the module. Modules should only change versions when the -upgrade flag is explicitly used.

[jcarlson]

@apparentlymart Most of the information that would be required for this "lock" file appears to be present in Terraform today in versions going back to at least 0.13.7 (I did not check older) in the project's .terraform/modules/modules.json file.

(root):main.tf

module "some-module" {
  source  = "app.terraform.io/my-org/my-module/aws"
  version = "~> 1.2"
}

my-module:main.tf

module "nested" {
  source  = "app.terraform.io/my-org/nested/aws"
  version = "~> 3.1"
}

(root):.terraform/modules/modules.json

{
  "Modules": [
    {
      "Key": "some-module",
      "Source": "app.terraform.io/my-org/my-module/aws",
      "Version": "1.2.3",
      "Dir": ".terraform/modules/some-module"
    },
    {
      "Key": "some-module.nested",
      "Source": "app.terraform.io/my-org/nested/aws",
      "Version": "3.3.0",
      "Dir": ".terraform/modules/some-module.nested"
    }
  ]
}

While it would be nice to have Terraform support this natively, if this modules.json file were included in source control, it might be enough information for a wrapper script to use to download and install the "locked" versions of each module...

[apparentlymart]

Hi @jcarlson!

What you have there is the modules analog to what Terraform did for providers before we introduced the dependency lock file: there was a hidden, implementation-detail-only JSON file under .terraform/plugins that captured what terraform init installed so that other commands could find it. The dependency lock file totally replaced that hidden file for providers. So indeed as you say I would expect work on this issue to entirely replace that modules.json file with equivalent metadata in .terraform.lock.hcl.

What isn't clear at this time is what exactly the lock file should be tracking in order to meet the use-cases people want it to meet.

One answer would be to copy the existing lock file exactly, identifying each module block individually by its path like some-module or some-module.nested and remembering its source address and, if appropriate, version number. We would need to figure out what to do with the Dir property that can currently vary depending on where you are running Terraform (if e.g. you set TF_DATA_DIR to specify a different location than .terraform), but providers already gave us a possible answer to that: we just rely on convention rather than explicit path saving, so that we can know from a provider's source address and our current platform which path under TF_DATA_DIR should contain the provider executable.

However, in the past other feedback has suggested that people want to lock modules on a per-source-address basis instead, which might lead to a formulation more like the following:

module "app.terraform.io/my-org/my-module/aws" {
  version = "1.2.3"
  hashes = [
    # ...
  ]
}

The idea in this case then would be that any module block that specifies source = "app.terraform.io/my-org/my-module/aws" would be required to match this constraint and hashes, which means that adding a new call to a module already known can rely on the existing hashes and thus get the "trust on first use" benefit of the entry already recorded.

There are some other variations too, such as treating the source address and the version number together as the lookup key and associating a set of hashes with each, so that there can still be multiple versions of the same module used in the same configuration but we can still share the hashes whenever two have the same version selected.

In order to move forward here we need to complete some product research to learn which of these is the best fit for the requirements. Our current structure where the manifest is only a temporary file can get away without this prior engineering because we are free to change it at any time and have terraform init just regenerate it if the format or structure has changed in some way, but once we have people fixing these values in their repositories they become subject to compatibility constraints and cannot be changed again, and so it's important to ensure we have a design that supports all of the needed future changes to Terraform's modules model, and not focus only on what the current design requires.

[jcarlson]

@apparentlymart well said, thank you for the detailed explanation!

I may still pursue a mechanism to consume that modules.json file in the short term; we are still on Terraform 0.13, and it will be a while before we get up to modern 1.2 or better.

Ultimately, what we're hoping to get out of this lock file is a way to detect, based on metadata, which of our many, many configurations need to be applied, because a dependency (either direct or transitive) has been updated.

We have, for a while, been using Git-based module source addresses in both our root configurations as well as our in-house modules, so any changes to dependencies could be identified explicitly through changes to source code. Now, however, we've switched to a module registry with version constraints, which means a configuration could have pending changes even though no code has changed.

So in summary, the three things we want to solve for are:

1. Avoid cascading updates to intermediate modules, e.g. root => Module A ~> 1.2 => Module B ~> 2.3 => Module C ~> 4.5. When Module C releases 4.6.0, it can precipitate a change in infrastructure on the root, but we do not want to have to update modules A and B in addition to the root; this would be roughly the same behavior as npm, yarn or bundler provide
2. Detect when transitive dependencies might create change in the root, e.g. identify that the root has "locked" a transitive dependency on Module C @ 4.5.1 but that 4.6.0 is available, thus the root should run terraform init -upgrade to resolve the newer version and run a plan
3. Ensure that production-time deployments resolve the same transitive dependencies that were resolved in non-production environments, e.g. the root resolves transitive dependency Module C @ 4.5.1 on Monday and then runs terraform apply on the dev workspace; on Tuesday, Module C releases 4.6.0; on Wednesday, the prod workspace is applied for the root configuration using an ephemeral CI/CD environment; the prod workspace should still resolve Module C @ 4.5.1 since that was what was used in the dev workspace earlier in the week

I'd be happy to help on this effort in anyway that I can; hopefully, providing some concrete uses cases is a good start.

[stevehipwell]

As an initial solution I'd like the lock file to capture the hash of modules and error if the hash doesn't match when running a command. I have some opinions on other parts of this discussion but for now this would solve a supply chain concern with the current implementation (e.g. re-tagging a module to provide a malicious payload).

I'm not sure if it's related but adding a new version_match parameter to module blocks would allow proper handling of modules which aren't hosted in a TF registry; basically the version can be extracted to the lock file using the regex (first group in this but could use lookahead/lookbehind instead). An example of it's use is below.

module "my_module" {
  source             = "https://mysite.io/modules/my-module-v1.0.0.tgz"
  version_match = "(?:v)(\d+\.\d+\d+)(?:\.tgz$)"

[parth995]

@redzioch Curious to know how to approach managing module version in case of multiple environments using same SSH Git URL format.

[tmccombs]

Using exact version constraints is indeed our recommended approach for now

The problem with this is that if you use a module that itself uses a module with a version constraint instead of an exact version, there is no way to lock the version. So you end up needing to have exact versions in all modules as well. And that in turn means if you make a minor bug fix, you have to propagate the version increase up through multiple dependencies.

Ideally, if I had a scenario where my root module A depends on module B and module B depends on module C, I could release a new version of module C, and then just update the lock file for module A to get the new version of C, without having to release a new version of module B.

[sergei-ivanov]

We are currently using a monorepo for our Terraform modules (200+ and growing), and we are using git URLs with specific tags in the projects that consume the modules. We are on our way to split that monorepo into smaller repos and use a private Terraform registry to publish the modules. And the anticipated problem with that approach is exactly as @jcarlson describes: we will either use version ranges and lose control over updates, or we will have to use specific versions and painstakingly propagate any changes up the dependency graph. Neither option is ideal. Our use case for version locking is also exactly as @jcarlson describes: we need to allow locking each module path separately, simply for the reason that we are dealing with some long-living projects (3+ years), and some instances of the modules have to be locked to older versions because newer versions may trigger resource recreation.

[verenion]

@sergei-ivanov We have just ran into this same limitation. Our use case is very similar - we have a private module for provisioning our kubernetes cluster, setting up firewalls, etc. Then we have one repository for each environment (production, staging, testing, etc.) - As we are currently using the module like this:

module "cluster" {
  source  = "app.terraform.io/xxx/xxx/xxx"
  version = "~> 1.0.0"

If someone changes the core module, all of those projects suddenly have drift - which is very unexpected to me.

[dosilyoun]

I am resolving my modules via a JFrog terraform module registry. However, I consider JFrog as an untrusted entity, so I would like to check at least the checksum for my modules as it is checked for providers. Do we have any development on going? This issue is getting referenced but didnt see any progress. As some of logic is already implemented for providers, how much work would it be for modules as well?

[leanrobot]

My team also would like to express interest in having support for exact module version in .terraform.lock.hcl. Similar to other use cases above, we are pinning the exact module version in it's instantiation as of now, since we don't have the ability to share a form of "locked" state across users who are running apply in our project.

I am in support of a solution which allows locking on a per-module instantiation basis, IE taking into account the modules name when resolving versions.

We are on terraform version 1.4.6

[boillodmanuel]

Hi @apparentlymart,

Any progress on this topic?

Regarding this:

However, in the past other feedback has suggested that people want to lock modules on a per-source-address basis instead, which might lead to a formulation more like the following:

• We must be able to use different version of same module in a single project, so don't lock per-source please.
• Today, this is not possible for provider (Support for running multiple versions of the same provider #16641). My opinion is that is should be possible for provider also. I don't see any reason to not allow creating a resource with aws@4 and another with aws@5. Moreover, having a single version leads to situation impossible to solve in a single project...

[lexfrei]

Any news?