terragrunt state pull throws an "unexpected end of JSON input" error

[cswingler]

I'm not sure what's going on here, but if you run:

% terragrunt init
<snip>
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
<snip>
Initializing provider plugins...

Terraform has been successfully initialized!
% terragrunt state pull > terraform.tfstate
[terragrunt] [/Users/chrisswingler/src/infra_terraform] 2018/07/06 12:03:12 Running command: terraform --version
[terragrunt] 2018/07/06 12:03:13 Reading Terragrunt config file at /Users/chrisswingler/Sprout/src/infra_terraform/terraform.tfvars
[terragrunt] 2018/07/06 12:03:13 unexpected end of JSON input
[terragrunt] 2018/07/06 12:03:13 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

Terragrunt pukes on something not being valid JSON. I don't know _what_ it's puking on, though. terraform state pull works fine.

Debug output:

➜  infra_terraform [default] git:(terragrunt_k8s) ✗  TERRAGRUNT_DEBUG=trye terragrunt state pull > terraform.tfstate
[terragrunt] [/Users/chrisswingler/Sprout/src/infra_terraform] 2018/07/06 13:16:51 Running command: terraform --version
[terragrunt] 2018/07/06 13:16:51 Reading Terragrunt config file at /Users/chrisswingler/Sprout/src/infra_terraform/terraform.tfvars
[terragrunt] 2018/07/06 13:16:51 *json.SyntaxError unexpected end of JSON input
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/remote/terraform_state_file.go:79 (0x142fbbb)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/remote/terraform_state_file.go:71 (0x142faed)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/remote/terraform_state_file.go:56 (0x142f925)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/remote/remote_state.go:69 (0x142c10b)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:571 (0x14d4182)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:441 (0x14d3361)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:410 (0x14d31d9)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:323 (0x14d27b6)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:235 (0x14d1a48)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:204 (0x14d1950)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:185 (0x14d1644)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:502 (0x14762c8)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:268 (0x14743ec)
/private/tmp/terragrunt-20180703-977-jbnzes/terragrunt-0.15.2/main.go:20 (0x14d7be4)
/usr/local/Cellar/go/1.10.3/libexec/src/runtime/proc.go:198 (0x102ccb2)
/usr/local/Cellar/go/1.10.3/libexec/src/runtime/asm_amd64.s:2361 (0x1056f11)

[terragrunt] 2018/07/06 13:16:51 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[brikis98]

Can you set the TERRAGRUNT_DEBUG env var to true to see if that gives more useful output?

[cswingler]

Updated post with debug output, thanks for the tip!

[brikis98]

Thanks. Based on the stacktrace, I think this is the code that's failing:

https://github.com/gruntwork-io/terragrunt/blob/master/remote/terraform_state_file.go#L78-L80

I think that implies your .tfstate file is not valid JSON... Could you take a look at it?

[cswingler]

This is just after an init and our remote state is stored in s3. It's executing much too quickly to seem like it's actually pulling it down:

➜  infra_terraform [default] git:(terragrunt_k8s) ✗  time terragrunt pull > terraform.tfstate
[terragrunt] [/Users/chrisswingler/Sprout/src/infra_terraform] 2018/07/06 14:57:15 Running command: terraform --version
[terragrunt] 2018/07/06 14:57:15 Reading Terragrunt config file at /Users/chrisswingler/Sprout/src/infra_terraform/terraform.tfvars
[terragrunt] 2018/07/06 14:57:15 Running command: terraform pull
[terragrunt] 2018/07/06 14:57:15 Hit multiple errors:
exit status 127
terragrunt pull > terraform.tfstate  0.51s user 0.13s system 107% cpu 0.598 total

If I download it with terraform state pull it is indeed valid json.

For what it's worth, our state is a little large (4.6 MB).

[brikis98]

Do you by chance have a .tfstate file sitting around in the current directory? E.g., if you run this all from a clean folder and with the --terragrunt-source-update flag, does it work?

[chrisportela]

I had this issue today and wanted to chime in that yes having *.tfstate files in the directory will cause problems. I ran in to it when I pulled remote state to terraform.tfstate and it started messing with everything.

I recommend using *.out or similar instead.

[brikis98]

Closing due to inactivity

[thirstydeveloper]

I also ran into this issue, following what seemed to me to be a fairly standard use case.

1. I started with a simple manifest and local state storage
2. I then migrated to using terragrunt to handle configuring remote state in S3
3. terragrunt init properly migrated the state to S3
4. terragrunt plan broke with the invalid JSON error
5. Removing the *.tfstate files from my local file system fixed the error

It would be helpful if terragrunt would either:

1. Offer to backup the local state file in a manner that allows terragrunt to run after migrating to remote state
2. Print a warning about local state files if remote state has been configured

@brikis98 - could this issue be reopened?

[brikis98]

@thirstydeveloper Reopened! Could you by any chance create a simple repro?

[thirstydeveloper]

@brikis98 - Here you go.

https://github.com/thirstydeveloper/terragrunt-s3-state-migration

Also of note, after changing terraform.tfvars to use s3 remote state, the first terragrunt apply seems to work (at least with this example), but all subsequent terragrunt apply produce the invalid JSON error.

[brikis98]

Thanks! We'll take a look.

[koka424]

I ran into this as well. Removing the terraform.tfstate file fixed the issue.

[jeffdyke]

I'm running into this currently and the only terraform.tfstate i have is 0 bytes b/c it is remotely managed in s3, if i remove it it wants to create and destroy everything. @thirstydeveloper or @koka424 - did the tfstate file have anything inside?

[jeffdyke]

I'm running into this currently and the only terraform.tfstate i have is 0 bytes b/c it is remotely managed in s3, if i remove it it wants to create and destroy everything. @thirstydeveloper or @koka424 - did the tfstate file have anything inside?

Edit: yes, removing the 0 byte file worked....it just didn't seem like it because i had other errors that this was masking. Which may speak to better messaging, like adding the filename. Clearly my mistake, and thanks for the great project.

[kalebwalton]

Ran into this today as well after initializing an artifactory state file as empty (following the prompts rather than copying my bad local state in).

UPDATE: It turned out to be a local issue... I removed my .terragrunt-cache and re-ran init, all is well. Not sure what happened...

[k3a]

Happened to me too (Terragrunt v0.23.10) after setting up S3 remote store backend.

The initialization asked for migration like this

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

and succeeded. The following attempt to do terragrunt plan resulted in

[terragrunt] 2020/04/25 00:32:37 unexpected end of JSON input

which was caused by an empty terraform.tfstate file in the cache. Removing the file fixed it.

[augustgerro]

Do you by chance have a .tfstate file sitting around in the current directory? E.g., if you run this all from a clean folder and with the --terragrunt-source-update flag, does it work?

@brikis98
Could you please add this as a warning to the stderr, I guess it will safe a lot of time for all of us.

[zwass]

I just ran into this. Seems like either deleting the empty file or issuing a warning would be a good fix.