terragrunt not copying doted files (.something) from working dir in temp dir

[ebarault]

this happens during stage labelled as

Copying files from my/workdir into /Users/ebarault/.terragrunt/qpryAuEVuRxDI-1s0Hw-004ArQc/WTycM2xCvnoeoxRFsIteSUwW1Zk

In my/workdir i have dotet files such .config, .env that are essential to my terraform task.

but when investigating after errors I noted that these files are not copied by terragrunt in the tmp dir Users/ebarault/.terragrunt/qpryAuEVuRxDI-1s0Hw-004ArQc/WTycM2xCvnoeoxRFsIteSUwW1Zk

is there a rational for this, or is this a bug?

i'm running terragrunt v0.13.23

[brikis98]

Terragrunt intentionally skips hidden files and folders: https://github.com/gruntwork-io/terragrunt/pull/236. Ideas on how to include them, while still skipping "problematic" ones (e.g., .terraform, .git, etc) are welcome!

[ebarault]

ah ah, the wonderful side effect ! This is deceptive.
this means one cannot trust the copy step as a carbon copy, so I should always pass the absolute path of any included src/assets/whatever ressources to my module.

on how to improve this : why not maintaining a black list of unwanted files instead of running a whitelist excluding all hidden files/folders ?

[brikis98]

Unintended consequences :)

Right now, Terragrunt doesn't copy any hidden files or folders. I'd be fine with either a blacklist or whitelist approach, so long as you could configure it. Example:

terragrunt = {
  terraform {
    source = "..."
    exclude_from_copy = [".terraform", ".git"]
  }
}

[rgarrigue]

Maybe some kind of .ebaraultignore ? :trollface: ;-)

[ebarault]

please forgive this funny colleague of mine ;-)
I'll try sparing some time soon to work on that one.

[brikis98]

Method that copies files/folders: https://github.com/gruntwork-io/terragrunt/blob/master/util/file.go#L136

Terragrunt config as it's parsed from file: https://github.com/gruntwork-io/terragrunt/blob/master/config/config.go#L32

Terragrunt config as it's represented in memory after parsing: https://github.com/gruntwork-io/terragrunt/blob/master/config/config.go#L20

[cha7ri]

It seems that Terragrunt copy just the terraform.tfvars and for the rest of the configuration files it uses terrafrom init to do that. So I think fixing this issue here well not be usefull unless terragrunt uses an other method to copy the configuration files.
Please check this issue from info.

[ebarault]

@cha7ri : ah good point, @brikis98 : is terragrunt using another type of copy mechanism ?

[brikis98]

Terragrunt uses init to check out the code specified in source and then copies files over from the local folder using the code in this comment: https://github.com/gruntwork-io/terragrunt/issues/394#issuecomment-350764413

[cha7ri]

Below I changed the function that copy the files and I added a debug message:

```func CopyFolderContents(source string, destination string) error {
files, err := ioutil.ReadDir(source)
if err != nil {
return errors.WithStackTrace(err)
}

for _, file := range files {
    logger := CreateLogger("")
    logger.Printf("ch7ri copying %s", file.Name())
    src := filepath.Join(source, file.Name())
    dest := filepath.Join(destination, file.Name())

    if file.IsDir() {
        if err := os.MkdirAll(dest, file.Mode()); err != nil {
            return errors.WithStackTrace(err)
        }

        if err := CopyFolderContents(src, dest); err != nil {
            return err
        }
    } else {
        if err := CopyFile(src, dest); err != nil {
            return err
        }
    }
}

return nil

}
```

The only output that I have is this message: ch7ri copying terraform.tfvars
FYI: the terraform.tfvars is not in the same folder as the configuration files.
I run the test using go run main.go plan. I tried --terragrunt-source-update the same output.

[brikis98]

What else do you have in the directory?

[lorengordon]

Using the new hooks feature to copy files that terragrunt does not has been working pretty well for me.

[CarltonHenderson]

Using the new hooks feature to copy files that terragrunt does not has been working pretty well for me.

Could you share specifics? A workaround would be good to have on this issue.