Terraform: Data sources with non-deterministic IDs show perpetual diff with Terraform 0.13.3

[srishtipmishra]

Terraform Version

0.13.3

Terraform Configuration Files

data "oci_file_storage_mount_targets" "mount_targets" {
  #Required
  availability_domain = data.oci_identity_availability_domain.ad.name
  compartment_id = var.compartment_ocid

  //depends_on = [ oci_file_storage_mount_target.mount_target ]
}

data "oci_file_storage_export_sets" "export_sets" {
  #Required
  availability_domain = data.oci_identity_availability_domain.ad.name
  compartment_id = var.compartment_ocid

  //depends_on = [oci_file_storage_export_set.export_set]
}

resource "oci_file_storage_mount_target" "mount_target" {
  #Required
  availability_domain = data.oci_identity_availability_domain.ad.name
  compartment_id      = var.compartment_ocid
  subnet_id           = oci_core_subnet.my_subnet.id
}

resource "oci_file_storage_export_set" "export_set" {
    #Required
    mount_target_id = oci_file_storage_mount_target.mount_target.id
    depends_on = [ oci_file_storage_mount_target.mount_target ]
}

Debug Output

Full debug output can be obtained by running Terraform with the environment variable TF_LOG=trace. Please create a GitHub Gist containing the debug output. Please do _not_ paste the debug output in the issue, since debug output is long.

Debug output may contain sensitive information. Please review it before posting publicly, and if you are concerned feel free to encrypt the files using the HashiCorp security public key.

Terraform will perform the following actions:

  # data.oci_file_storage_export_sets.export_sets will be read during apply
  # (config refers to values not yet known)
 <= data "oci_file_storage_export_sets" "export_sets"  {
        availability_domain = "NyKp:PHX-AD-1"
        compartment_id      = "ocid1.compartment.oc1..aaaaaaaa4rv5j2vzbrwaztnzvtu7kgswtigms4llcbylelylsqt2l3kl7gaa"
        export_sets         = [
            {
                availability_domain = "NyKp:PHX-AD-1"
                compartment_id      = "ocid1.compartment.oc1..aaaaaaaa4rv5j2vzbrwaztnzvtu7kgswtigms4llcbylelylsqt2l3kl7gaa"
                display_name        = "mt-8254 - export set"
                id                  = "ocid1.exportset.oc1.phx.aaaaacvippxoyzt3obuhqllqojxwiotqnb4c2ylefuzaaaaa"
                max_fs_stat_bytes   = ""
                max_fs_stat_files   = ""
                mount_target_id     = ""
                state               = "ACTIVE"
                time_created        = "2020-09-18 18:53:36.17 +0000 UTC"
                vcn_id              = "ocid1.vcn.oc1.phx.amaaaaaajcdxrhqa5ju3y4temaqcmijnhbbp74kjx6tygsrb5b767a7kz5jq"
            },
            {
                availability_domain = "NyKp:PHX-AD-1"
                compartment_id      = "ocid1.compartment.oc1..aaaaaaaa4rv5j2vzbrwaztnzvtu7kgswtigms4llcbylelylsqt2l3kl7gaa"
                display_name        = "export set for mount target 2"
                id                  = "ocid1.exportset.oc1.phx.aaaaaby27vfb4qhsobuhqllqojxwiotqnb4c2ylefuzaaaaa"
                max_fs_stat_bytes   = ""
                max_fs_stat_files   = ""
                mount_target_id     = ""
                state               = "ACTIVE"
                time_created        = "2020-08-31 19:31:27.404 +0000 UTC"
                vcn_id              = "ocid1.vcn.oc1.phx.amaaaaaajcdxrhqarvakj6dyyxit5m2mrcdhl6l6isxfen7xgm4ori7auywa"
            },
            {
                availability_domain = "NyKp:PHX-AD-1"
                compartment_id      = "ocid1.compartment.oc1..aaaaaaaa4rv5j2vzbrwaztnzvtu7kgswtigms4llcbylelylsqt2l3kl7gaa"
                display_name        = "displayName2 - export set"
                id                  = "ocid1.exportset.oc1.phx.aaaaacvippxmpxbjobuhqllqojxwiotqnb4c2ylefuzaaaaa"
                max_fs_stat_bytes   = ""
                max_fs_stat_files   = ""
                mount_target_id     = ""
                state               = "ACTIVE"
                time_created        = "2020-06-16 02:44:09.658 +0000 UTC"
                vcn_id              = "ocid1.vcn.oc1.phx.amaaaaaajcdxrhqamsl36aecluwczehczc5hbxgmk4f65l4hbyi5nhnhfuiq"
            },
        ]
      ~ id                  = "2020-09-21 17:11:31.340791 +0000 UTC" -> "2020-09-21 17:11:32.673724 +0000 UTC"
    }

  # data.oci_file_storage_mount_targets.mount_targets will be read during apply
  # (config refers to values not yet known)
 <= data "oci_file_storage_mount_targets" "mount_targets"  {
        availability_domain = "NyKp:PHX-AD-1"
        compartment_id      = "ocid1.compartment.oc1..aaaaaaaa4rv5j2vzbrwaztnzvtu7kgswtigms4llcbylelylsqt2l3kl7gaa"
      ~ id                  = "2020-09-21 17:11:31.42748 +0000 UTC" -> "2020-09-21 17:11:32.674163 +0000 UTC"
        mount_targets       = [
            {
                availability_domain = "NyKp:PHX-AD-1"
                compartment_id      = "ocid1.compartment.oc1..aaaaaaaa4rv5j2vzbrwaztnzvtu7kgswtigms4llcbylelylsqt2l3kl7gaa"
                defined_tags        = {}
                display_name        = "mt-8254"
                export_set_id       = "ocid1.exportset.oc1.phx.aaaaacvippxoyzt3obuhqllqojxwiotqnb4c2ylefuzaaaaa"
                freeform_tags       = {}
                hostname_label      = ""
                id                  = "ocid1.mounttarget.oc1.phx.aaaaacvippxoyzt4obuhqllqojxwiotqnb4c2ylefuzaaaaa"
                ip_address          = ""
                lifecycle_details   = ""
                nsg_ids             = []
                private_ip_ids      = [
                    "ocid1.privateip.oc1.phx.abyhqljsxmyvuxhkeiqf3orkfz2x4jaunuhe4hke5hlnrtqqm4idcfuu3p3a",
                ]
                state               = "ACTIVE"
                subnet_id           = "ocid1.subnet.oc1.phx.aaaaaaaacheszpqwu22ns7xk2a2hhfafcecn3efvhexnkq3vbv36ch7gb6iq"
                time_created        = "2020-09-18 18:53:36.17 +0000 UTC"
            },
            {
                availability_domain = "NyKp:PHX-AD-1"
                compartment_id      = "ocid1.compartment.oc1..aaaaaaaa4rv5j2vzbrwaztnzvtu7kgswtigms4llcbylelylsqt2l3kl7gaa"
                defined_tags        = {
                    "example-tag-namespace-all.example-tag" = "value"
                }
                display_name        = "my_mount_target_2"
                export_set_id       = "ocid1.exportset.oc1.phx.aaaaaby27vfb4qhsobuhqllqojxwiotqnb4c2ylefuzaaaaa"
                freeform_tags       = {
                    "Department" = "Accounting"
                }
                hostname_label      = ""
                id                  = "ocid1.mounttarget.oc1.phx.aaaaaby27vfb4qhtobuhqllqojxwiotqnb4c2ylefuzaaaaa"
                ip_address          = ""
                lifecycle_details   = ""
                nsg_ids             = [
                    "ocid1.networksecuritygroup.oc1.phx.aaaaaaaa4toe7ubaw7cntqyripqgybgw7xgf4pxyt4jvgepznpyc7jmgce6q",
                ]
                private_ip_ids      = [
                    "ocid1.privateip.oc1.phx.abyhqljs634ti4yzxob7sd3bwkhdcrcr2xb4daw2fqhssdlijbcoha4ydzjq",
                ]
                state               = "ACTIVE"
                subnet_id           = "ocid1.subnet.oc1.phx.aaaaaaaaefne4gi46fc5iyg6yd2ohd3bfqo4dhemgmac4h6noojvoyifu7pq"
                time_created        = "2020-08-31 19:31:27.404 +0000 UTC"
            },
            {
                availability_domain = "NyKp:PHX-AD-1"
                compartment_id      = "ocid1.compartment.oc1..aaaaaaaa4rv5j2vzbrwaztnzvtu7kgswtigms4llcbylelylsqt2l3kl7gaa"
                defined_tags        = {
                    "example-tag-namespace-all.example-tag" = "updatedValue"
                }
                display_name        = "displayName2"
                export_set_id       = "ocid1.exportset.oc1.phx.aaaaacvippxmpxbjobuhqllqojxwiotqnb4c2ylefuzaaaaa"
                freeform_tags       = {
                    "Department" = "Accounting"
                }
                hostname_label      = ""
                id                  = "ocid1.mounttarget.oc1.phx.aaaaacvippxmpxbkobuhqllqojxwiotqnb4c2ylefuzaaaaa"
                ip_address          = ""
                lifecycle_details   = ""
                nsg_ids             = []
                private_ip_ids      = []
                state               = "FAILED"
                subnet_id           = "ocid1.subnet.oc1.phx.aaaaaaaahkgp57yeetknqlqvgrv6rvlfxqdhye6wndcsvmecwxlftq2pszra"
                time_created        = "2020-06-16 02:44:09.658 +0000 UTC"
            },
        ]
    }

Plan: 0 to add, 0 to change, 0 to destroy.

Crash Output

Expected Behavior

No diffs expected onterraform plan

Actual Behavior

Datasource ids show a diff on plan after successful apply

Steps to Reproduce

Please list the full steps required to reproduce the issue, for example:

1. terraform init
2. terraform apply
3. terraform plan

Additional Context

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

• #25812

[jbardin]

Hi @srishtipmishra,

Thanks for filing the issue. Would it be possible to supply a stand-alone reproduction case for one of these data sets?

What is likely happening is that there is an apparent change in the data source configuration, triggering it to be read again. The data source then returns the original data, except for the ID field which shows up as an update. This means the issue could be that the data source fields are inconsistent with the config, or an input to the data source is inconsistent, but the changing ID is not the problem.

There may also be some WARN messages in the trace logs along with these data sources or resources which might give a clue into what is happening.

[srishtipmishra]

Hi @jbardin - The zip is larger than 10MB. Is there a better way of sharing it? Fyi, the root cause is known here. Our datasource ID is a non-static one (timestamp). So, every time a read happens new value is set in SetData() which is causing the diff. We also tried changing it to a static value on provider's side and the diff goes away. However, this is hitting backward compatibility for us since issue appears only in TF13. Let me know if you need any other data my end.

[jbardin]

I don't think the full logs will help in this case, the warning messages are about all that's useful for this situation.

The non-static ID is not the root cause here; ID isn't a special field for Terraform at all, and it would be just as valid for any other field to change on every read and produce the same result. Removing unnecessary changes from the data source will of course help the situation, but the source of the problem is that there appears to be a change in the configuration causing the data source to be read again. If there is no apparent change in configuration during plan, then the data source is not read again, and there is no chance for a new ID to show up as a change.

This will all hopefully be a non-issue in the upcoming 0.14 release, which will eliminate the separate Refresh phase when creating a plan. Once that's in place, we should only have to read a data source once in a plan+apply cycle, and won't see any updates that should have been transparent to the user.

[honcharenkoy90]

Hi, all
Is this issue resolved in 0.13.14?

[jbardin]

Since there isn't any more we can do here from the core side, I'm going to close this out for now.
The mentioned changes for 0.14 have all been merged, and the handling of data sources has been further improved, so I suspect that the cause for the data sources being re-read will be suppressed.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.