Sourcing module with git by cloning in shallow mode

[samber]

Hi there,

We have a heavy git repository with a lot of Terraform modules. When we execute a terraform get it take ~20min to download dependencies (a git clone for each module call !).

We removed from the repository some binary files on the master branch. But unfortunately, nothing changed because Terraform do a git clone instead of git clone --depth 1.

[in4mer]

Related #14036

[clumsysnake]

also having this issue.

[scottconnerly]

Related #19277

[czerasz]

Currently using depth=1 together ref parameter will not always work.

Example main.tf:

module "test" {
  source="git::[email protected]:czerasz/terraform-aws-container-override.git?depth=1&ref=tags/v0.1.0"
}

The terraform init for this configuration returns:

...
Error: Failed to download module

Could not download module "test" (main.tf:1) source code from
"git::[email protected]:czerasz/terraform-aws-container-override.git?depth=1&ref=tags/v0.1.0":
error downloading
'ssh://[email protected]/czerasz/terraform-aws-container-override.git?depth=1&ref=tags%2Fv0.1.0':
/usr/bin/git exited with 1: error: pathspec 'tags/v0.1.0' did not match any
file(s) known to git.

This is because Terraform (through the hashicorp/go-getter library) does the following:

• clone the repo with the --depth 1 flag:

  /usr/bin/git clone --depth 1 ssh://[email protected]/czerasz/terraform-aws-container-override.git .terraform/modules/test

• then it tries to checkout the 0.1.0 tag:

  /usr/bin/git checkout tags/v0.1.0

The problem is that by using --depth 1 git does NOT "see" all available tags/branches:

$ cd .terraform/modules/test 
$ git tag | cat 
v0.1.1

[in4mer]

Please correct me if I'm wrong, but your question doesn't make sense.. tags are treeish objects. You've checked out a shallow copy of the repo, meaning you have no chunks besides those used to comprise the currently specified commit you've checked out. So in essence, that means you have no history besides "now."

Ah, yeah, I see. You've asked for tag=123 with a depth=1, and gotten tag=125, and of course no other tags. Have you tried reversing the URL query fields, just as a test?

[czerasz]

@in4mer I tried reversing the fields, but it's not relevant.

The issue is, as You mentioned, there is no history besides "now."

[zachwhaley]

Could the clone command just add --branch to its arguments?

https://www.git-scm.com/docs/git-clone#Documentation/git-clone.txt--bltnamegt

[zachwhaley]

I opened a PR to fix this, although I'm not sure if I was supposed to open it for go-getter or terraform 😕

[trallnag]

Would be definitely nice to see this. Quite a bummer that using depth in combination with ref is not working.

[carbonrobot]

Have we tried using the full git refspec for this? Using a ref of tags/v0.1.0 is only a short version of the a refspec. The equivalent is git fetch --depth 1 origin refs/tags/1.1:refs/tags/1.1, so it might be worth trying to use the full version.

This is explained in the git manual

--depth=
Limit fetching to the specified number of commits from the tip of each remote branch history. If fetching to a shallow repository created by git clone with --depth= option (see git-clone[1]), deepen or shorten the history to the specified number of commits. Tags for the deepened commits are not fetched.

Using the full refs/* path however, will pull the commit that is referenced.

https://stackoverflow.com/a/26619302/118224

[sebglon]

With terraform we can't specify the full refs/* AS proposed by @carbonrobot
We have exactly the same issue

[sebglon]

But this PR may fix this bug: hashicorp/go-getter#266 or hashicorp/go-getter#266

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.