Terragrunt: Issues while running tests on Windows

[marcoreni]

This issue is to keep track of the errors encountered while running tests on Windows

1. Filename too long

--- FAIL: TestLocalWithRelativeExtraArgsWindows (2.42s)
    integration_test.go:1205: Failed to run Terragrunt command 'terragrunt apply --terragrunt-non-interactive --terragrunt-working-dir fixture-download/local-relative-extra-args-windows' due to error: Hit multiple errors:
        exit status 1
Error downloading modules: Error loading modules: error downloading 'https://github.com/gruntwork-io/terragrunt.git?ref=v0.9.9': C:\[...]\git.exe exited with 1: D:/development/gocode/src/github.com/gruntwork-io/terragrunt/test/fixture-download/local-relative-extra-args-windows/.terragrunt-cache/MP9kZ_1zQFGF487Ix2mJfSnDVpk/W0XniauQ2qGRGzjX0cm-oZhY_ZQ/.terraform/modules/4cd1a0cd655aa737c4bdb93cdaf9af3b/.git/: Filename too long

1. Missing commands (also happens with touch and cp)

[terragrunt] 2018/10/01 21:41:29 Error running hook after_hook_1 with message: exec: "touch": executable file not found in %PATH%
--- FAIL: TestTerragruntInitHookWithSourceNoBackend (1.41s)
    integration_test.go:1215: Copying fixture-hooks/init-once to C:\Users\reni\AppData\Local\Temp\terragrunt-test131535916
    integration_test.go:159: Did not expect to get error: Hit multiple errors:
        Hit multiple errors:
        exec: "echo": executable file not found in %PATH%

[brikis98]

Thx for reporting. We unfortunately have not had time to set up CI on a Windows server, so these sorts of issues keep slipping in 😞

[DefSol]

I have hit the first issue which is a result of the maximum character length in a windows shell and the new .terragrunt-cache location. The issue is the same as running earlier versions of npm on windows. The path that terragrunt copies module information into is too long for the windows shell.

This is somewhat non-deterministic as it depends on two things

1. How deep in your files system you have the repo that contains your terragrunt files.
2. How many Terraform modules you pull in

As an example

The repo is cloned to

c:\repo\f1\f2\service-name

where f1 & f2 are some meaningful folder names
Our infra uses the following structure to follow a convention we have for remote state outputs.

service-name\infra\env\vpc\service-name\terraform.tfvars

This also pulls in a Terraform resource which also references another Terraform module.

When doing a plan, Terragrunt tries to copy artifacts to the .terragrunt-cache folder which adds what I presume is the hash of the files as the folder name. Eventually, it will exceed the maximum length of the windows shell as shown below

❯ terragrunt plan
[terragrunt] [C:\repo\f1\f2\service-name\infra\env\vpc\service-name] 2019/02/03 14:50:19 Running command: terraform --version
[terragrunt] 2019/02/03 14:50:23 Reading Terragrunt config file at C:/repo/f1/f2/service-name/infra/env/vpc/service-name/terraform.tfvars
[terragrunt] 2019/02/03 14:50:23 Terraform files in C:/repo/f1/f2/service-name/infra/env/vpc/service-name/.terragrunt-cache/5Tdn4oFgWTBz9zzBFW2At6IWGLY/NzsnV9yiqZAPa6SkkdDOTgA9O9g/service-name are up to date. Will not download again.
[terragrunt] 2019/02/03 14:50:23 Copying files from C:/repo/f1/f2/service-name/infra/env/vpc/service-name into C:/repo/f1/f2/service-name/infra/env/vpc/service-name/.terragrunt-cache/5Tdn4oFgWTBz9zzBFW2At6IWGLY/NzsnV9yiqZAPa6SkkdDOTgA9O9g/service-name
[terragrunt] 2019/02/03 14:50:23 Setting working directory to C:/repo/f1/f2/service-name/infra/env/vpc/service-name/.terragrunt-cache/5Tdn4oFgWTBz9zzBFW2At6IWGLY/NzsnV9yiqZAPa6SkkdDOTgA9O9g/service-name
[terragrunt] [C:\repo\f1\f2\service-name\infra\env\vpc\service-name] 2019/02/03 14:50:23 Initializing remote state for the s3 backend
[terragrunt] [C:\repo\f1\f2\service-name\infra\env\vpc\service-name] 2019/02/03 14:50:24 Running command: terraform init -backend-config=key=vpc/service-name/terraform.tfstate -backend-config=region=ap-southeast-2 -backend-config=encrypt=true -backend-config=dynamodb_table=terraform-locks -backend-config=bucket=f2nz-fp-remote-state-storage -lock-timeout=20m
Initializing modules...
- module.route53
  Getting source "git::ssh://[email protected]/org/terraform-route53.git?ref=1.1.0"
Error downloading modules: Error loading modules: error downloading 'ssh://[email protected]/org/terraform-route53.git?ref=1.1.0': C:\Program Files\Git\cmd\git.exe exited with 128: Cloning into '.terraform\modules\5a8b20e7b4c74ad33fa8c04390ada5fd'...
fatal: cannot write keep file 'C:/repo/f1/f2/service-name/infra/env/vpc/service-name/.terragrunt-cache/5Tdn4oFgWTBz9zzBFW2At6IWGLY/NzsnV9yiqZAPa6SkkdDOTgA9O9g/service-name/.terraform/modules/5a8b20e7b4c74ad33fa8c04390ada5fd/.git/objects/pack/pack-0ba3e00e61e71a5ee669c05c62dbd162842c3fdf.keep': Filename too long
fatal: index-pack failed

[terragrunt] 2019/02/03 14:50:28 Hit multiple errors:
exit status 1

Folders have been renamed

Work Arounds

• (Credit @lorengordon) export the env TERRAGRUNT_DOWNLOAD=C:.terragrunt-cache in my shell/profile.
• I managed to move the service name folder to the root of my c drive; luckily the dir structure was able to be traversed.
• Use a terragrunt container wrapped in a ps function that would perform the terragrunts

This issue is likely to become more apparent, as was the case for npm, and the second workaround may not even work.

[brikis98]

What's the proper fix? Is there some short version of file paths that Windows supports (with ~ characters substituting part of the path)?

[lorengordon]

Far as I know, there isn't much of a fix. I run into the same problem with a number of tools, even after turning on the LongPathsEnabled feature.

As a workaround for terragrunt, I export the env TERRAGRUNT_DOWNLOAD=C:\.terragrunt-cache in my shell/profile.

Another workaround is to use the Windows Subsystem for Linux (WSL).

[brikis98]

Hm, that's a shame. Thanks for sharing your workarounds, as hopefully they will help others!

[DefSol]

@lorengordon that's elegant, I did not think of that. This mimics the previous behaviour of storing the cache in the temp directory. I have confirmed it works on my failing scenario. Thanks!

What I like the most is that it doesn't affect my flow!

I recommend @lorengordon approach be the primary option for solving (giving everyone the best chance to be the limits in windows) this issue.

[jakauppila]

What would be the preference for CI on Windows? AppVeyor?

[brikis98]

Yup, I think AppVeyor would work!

[jakauppila]

FYI, came across this issue on the Terraform repo: https://github.com/hashicorp/terraform/issues/21173

[ignatz42]

I'm struggling with #1299 and have a couple of questions. Why does the terragrunt cache path need 73 characters? Here's an example of the long paths I'm seeing post TF 13.3 upgrade:

C:\src\vsts\myInstance\aProject\myTerraForm\experiments\UNIT\b6446abb.terragrunt-cache\LXqET6pD6BEswcFIg1rsjPV8ly4\0xE514VLijAM3D_w8dtQ73Wvxys.terraform\plugins\registry.terraform.io\hashicorp\azuread\1.0.0\windows_amd64\terraform-provider-azuread_v1.0.0_x5.exe (264 characters)

Until TF fixes the path limitation, would be possible to s/.terragrunt-cache/.tg/? (saves 14 characters)

Are the two random 27 character strings needed for the nested sub-directories? On my machine, the first directory is empty so is it really needed?

[Ettery]

While I find the Windows limitation frustrating, I agree with @ignatz42: is it necessary for terragrunt to add 74 chars to the path? (In my case .terragrunt-cache\-mwm0HJC1e8KJeyVKPJWYz94uN4\vQcOrMLBOtnMKIDMdOcs90oHoiA). The first folder always seems to be empty and even without the Windows issue it makes for a lot of folder navigation when debugging.