Terraform: CentOS 7 terraform init fails on sourcing module due to DNS resolution error

[flickerfly]

In short, terraform init fails when I run it on a CentOS 7 box with a module in git because it is appending a colon to the DNS lookup. If I modify the source to not have a colon immediately following the DNS name it works properly on CentOS. If I leave it that way it fails (as I expect it should) on MacOS. I've seen this in multiple terraform code directories setup in this same way.

Terraform Version

$ terraform -v
Terraform v0.11.7
+ provider.aws v1.29.0
+ provider.template v1.0.0

Terraform Configuration Files

module "environment" {
  source    = "git::ssh://[email protected]:/site/infrastructure-modules.git//modules/environment?ref=v0.3"
}

Debug Output

Using TF_LOG=trace didn't seem to output anything extra.

Crash Output

I've verified with tcpdump that CentOS is indeed trying to do a dns lookup on the name "site.git.beanstalkapp.com:". Notice the extra colon in the DNS name.

$ terrainit
Initializing modules...
- module.environment
  Getting source "git::ssh://[email protected]:/site/infrastructure-modules.git//modules/environment?ref=v0.3"
Error downloading modules: Error loading modules: error downloading 'ssh://git::ssh://[email protected]:/site/infrastructure-modules.git?ref=v0.3': /usr/bin/git exited with 128: Cloning into '.terraform/modules/d5ea407bf45b4c296aeb1af21b4b178f'...
ssh: Could not resolve hostname site.git.beanstalkapp.com:: Name or service not known
fatal: Could not read from remote repository.

Expected Behavior

DNS lookup should have succeeded because the extra colon on the end would be properly stripped from the name before lookup.

Actual Behavior

DNS lookup fails because it is malformed.

Steps to Reproduce

1. Be on a CentOS 7 system
2. Have a module sourced from a git repo
3. terraform init

Additional Context

I am passing some -backend-config variables for bucket, dynamodb_tables and region for the S3 backend.

I am not experiencing any DNS resolution problems anywhere else on the system.

[flickerfly]

I tried to toss in a domain bsa.com instead of the multi-subdomain stuff we have to deal with. This resulted in the same problem so it doesn't seem to be the multi-subdomain issue.

Error downloading modules: Error loading modules: error downloading 'ssh://[email protected]:/site/infrastructure-modules.git?ref=v0.3': /usr/bin/git exited with 128: Cloning into '.terraform/modules/1c4d9ceda24b495fc8864612be6c9abe'...
ssh: Could not resolve hostname bsa.com:: Name or service not known

[apparentlymart]

Hi @flickerfly! Sorry for this confusing behavior.

The interactions here are tricky because there are a number of different systems processing that source string. I _believe_ the processing is as follows:

1. Terraform itself sees that special git:: prefix and determines that the rest of the string is to be handled by git. (Terraform also trims off the part after and including // as a sub-path, and the ?ref=0.3 pseudo-querystring.
2. Terraform runs git clone followed by the string after the git:: prefix has been trimmed off, verbatim.
3. Git's own "pseudo-URL" parser then interprets it.
4. If the clone succeeds, Terraform then runs git checkout 0.3 due to the pseudo-querystring extracted in the first step.

If I've followed that thread correctly, I believe the problem is happening in the third step. Referring to the documentation of "Git URLs", I see that there are two different variants for SSH:

• ssh://[user@]host.xz[:port]/path/to/repo.git/
• [user@]host.xz:path/to/repo.git/

It looks like your URL here is mixing both of these forms together. I believe git is interpreting it as the first form, while the string you have after ssh:// there is actually in the second form. The first form uses a slash to separate the hostname from the path, rather than a colon.

With all of that said, I think the following form should work:

    source = "git::ssh://[email protected]/site/infrastructure-modules.git//modules/environment?ref=v0.3"

Here I just removed the colon to make it match the form shown for the ssh:// URL scheme.

Could you give that a try and let me know if it works better? Thanks!

[flickerfly]

Excellent explanation, Thank you! :-)

I was getting reports from Mac users that this didn't work. I found that as a solution, but didn't have the depth to show why it was valid, so I thought it wasn't.

I just tested it myself and it does appear to work on my machine so perhaps I need to investigate the problem they are having more thoroughly.

[apparentlymart]

Hi @flickerfly,

Funnily enough @nallanisai just said the same thing over in #18541, which was spookily opened just one day before your issue here.

I wonder if the version of git in the Xcode command line tools is an older version that didn't support this form of the SSH address syntax. Maybe using the older "scp-like" syntax would have better compatibility:

    source = "git::[email protected]:site/infrastructure-modules.git//modules/environment?ref=v0.3"

With the above, Terraform will run this:

git clone [email protected]:site/infrastructure-modules.git

I don't have a Mac here myself to test it so I can't confirm this. If you'd be able to give this form a try with the Mac users you mentioned that'd be very useful! If that works, we can update our docs to specifically caution against the other form.

Out of curiousity: we got three issues in quick succession about this after hearing nothing about it for years, so we were trying to guess why that might be. Did you base what you originally shared on an example you found somewhere? If it was in some of Terraform's official docs, we'd love to correct it!

[apparentlymart]

Hi again, @flickerfly! Sorry for the long silence.

We did eventually get to the bottom of this, and it turned out to be that Terraform can't handle this mixture of URL-like syntax and scp-like syntax at once, because it's ambiguous. If you use the URL-like form with ssh: at the front then you need to use slashes to separate the host from the path, because a colon would introduce a port number. The URL-like and scp-like forms are both supported, but the mixture is not:

  # URL-like
  source = "git::ssh://[email protected]/site/infrastructure-modules.git//modules/environment?ref=v0.3"

  # scp-like
  source = "git::[email protected]:site/infrastructure-modules.git//modules/environment?ref=v0.3"

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.