Terraform: Data sources in v0.13.0 now show additional values during apply, some of them sensitive

[stellirin]

Terraform Version

Terraform v0.13.0
+ provider registry.terraform.io/hashicorp/google v3.33.0
+ provider registry.terraform.io/hashicorp/google-beta v3.33.0
+ provider registry.terraform.io/hashicorp/http v1.2.0
+ provider registry.terraform.io/hashicorp/kubernetes v1.12.0
+ provider registry.terraform.io/hashicorp/null v2.1.2
+ provider registry.terraform.io/hashicorp/random v2.3.0
+ provider registry.terraform.io/hashicorp/vault v2.12.2

Terraform Configuration Files

data "google_storage_object_signed_url" "keys" {
  bucket = "my-bucket"
  path   = "my-file.txt"
  credentials = base64decode(google_service_account_key.signed_url.private_key)
}

Expected Behavior

Terraform v0.12.xx did not show data on apply:

  # data.google_storage_object_signed_url.keys will be read during apply
  # (config refers to values not yet known)
 <= data "google_storage_object_signed_url" "keys"  {
      + bucket      = "my-bucket"
      + credentials = (sensitive value)
      + id          = (known after apply)
      + path        = "my-file.txt"
      + signed_url  = (known after apply)
    }

Actual Behavior

Terraform v0.13.0 shows data on apply:

  # data.google_storage_object_signed_url.keys will be read during apply
  # (config refers to values not yet known)
 <= data "google_storage_object_signed_url" "keys"  {
        bucket      = "my-bucket"
        credentials = (sensitive value)
        duration    = "1h"
        http_method = "GET"
      ~ id          = "oldstring" -> "newstring"
        path        = "my-file.txt"
      ~ signed_url  = "https://storage.googleapis.com/oldurl" -> "https://storage.googleapis.com/newurl"
    }

Steps to Reproduce

1. terraform init
2. terraform apply

Additional Context

We see this with all data source types, where previously data was hidden by (known after apply) but now shows actual values. in many cases the values are sensitive values. This is most serious for vault secrets, where the output now looks like:

  # data.vault_generic_secret.vault_secret will be read during apply
  # (config refers to values not yet known)
 <= data "vault_generic_secret" "vault_secret"  {
        data             = {
            "password" = "my-password"
            "username" = "my-username"
        }
        data_json        = jsonencode(
            {
                password = "my-password"
                username = "my-username"
            }
        )
      ~ id               = "71646c5a-cfdb-a2a8-bf06-437ff4227117" -> "269340dc-7949-017e-f388-eb172ff9f0a9"
        lease_duration   = 0
        lease_renewable  = false
        lease_start_time = "RFC4449"
        path             = "secret/my-secret"
        version          = -1
    }

[stellirin]

We love Terraform, but we're looking to know if this change in behaviour is a bug, or it is expected and is the new normal going forward. This is especially important for the Vault provider, where we may have to investigate an alternative for secret storage.

I didn't see any note of this change in the changelog.

[jbardin]

Hi @stellirin,

Sorry that the output here is causing some trouble. Unfortunately not every change can be reasonable listed or easily described in the CHANGELOG, and in this case there wasn't a change in the output handling of data sources, but it is a result of enhanced handling of data sources in 0.13. It appears that #24904 was dropped from the final abridged CHANGELOG.

The reason (known after apply) was always shown 0.12, is that data sources could not be "planned" in any sense. We either had to read them during refresh (which was mostly silent), or delay their handling until apply. Now that there is a mechanism to evaluate data sources during plan, there is more information available to display in the plan output. In fact, showing as much information as possible when data sources change was a requested feature that this happened to satisfy as well.

In this case I think the resolution here will be to file issue with the associated provider, and request that the fields be marked as "sensitive" in the provider schema rather than relying on the information not being available most of the time. This of course won't prevent the data from showing up when referenced by other resources, but that has always been the case.

[stellirin]

Hi @jbardin,

Thanks for the detailed response. The mentioned change does seem to be the issue, and is probably going to affect any resource with an equivalent data source that do not have schemas fully in sync. I see that this is the case for the vault generic_secret.

[danieldreier]

I'm going to close this, because @jbardin explained it, and I do think that this is working as designed. I appreciate you bringing it up, and I've shared this with the ecosystem team so they know to expect more requests to update schemas in providers.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.