Terraform-provider-azurerm: Have azurerm_storage_blob updates on etag/hash changes

[arichiardi]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

The AWS s3_bucket_object has a wonderful feature for forcing a new resource - etag.
It would be awesome if azurerm_storage_blob could do the same, at the moment the only way to have a new resource is to have different file names.

New or Affected Resource(s)

• azurerm_storage_blob

Potential Terraform Configuration

```hcl
resource "azurerm_storage_blob" "artifact_blob" {
name = "${basename(var.artifact_path)}"
etag = "${md5(file("path/to/file"))}" <- desiderata

...

type = "block"
source = "${var.artifact_path}"
}

[mb290]

Can we just leverage the additional field Content-MD5 https://github.com/Azure/azure-sdk-for-go/blob/master/storage/blob.go#L387

[tombuildsstuff]

@mb290 looks like it, we should be able to make a new field in the Schema which is Optional + Computed + ForceNew and use that for detection, so a user could specify content_md5="${md5("file content")} to ensure this matches - what do you think?

[mb290]

@tombuildsstuff "should" and "reality" have turned into different things! I'm working through the uploads and struggling through the following error:

• azurerm_storage_blob.testsb: Error creating storage blob on Azure: Error updating block list for source file "text.txt": storage: service returned error: StatusCode=400, ErrorCode=InvalidMd5, ErrorMessage=The MD5 value specified in the request is invalid. MD5 value must be 128 bits and base64 encoded.

I'll report back with an update as I either succeed or fail miserably.

[tombuildsstuff]

@mb290 👍 cool - it may be worth opening an issue on the Azure SDK for Go repository for more clarification? Whilst that SDK is deprecated, unfortunately we're still blocked from migrating to the other Go Storage SDK

[mb290]

@tombuildsstuff I've tracked down the problem and the best explanation I can find of it so far is here http://cloudway.io/post/base64-encoded-128-bit/. Long story short, Terraform is generating the md5 hash in hex and Azure needs the MD5 hash in binary and then base64 encode that. I've been able to prove this out by manually generating the hash and uploading the file.

Any thoughts on how to proceed? One option is I modify the content_md5 to be auto generated in the resource which would take the file contents dynamically and run do the md5 conversion to binary as follows (taken from that URL):

cat text.txt | openssl dgst -md5 -binary | openssl enc -base64

[mb290]

@tombuildsstuff I've managed to come up with what I believe is probably the right solution. I've written a new function in Terraform core called base64md5 which is very similar to base64sha256. What this does is convert an entry to md5 binary and then encodes it with base64. This results in being able to use the blob resource as follows:

resource "azurerm_storage_blob" "testsb" {
  name = "text.txt"
  content_md5 = "${base64md5(file("text.txt"))}"

  resource_group_name      = "RG"
  storage_account_name   = "${azurerm_storage_account.test.name}"
  storage_container_name = "${azurerm_storage_container.test.name}"

  source = "text.txt"
  type = "block"
}

A snippet of the code for terraform 0.11.10 is:

func interpolationFuncBase64Md5() ast.Function {
    return ast.Function{
        ArgTypes:   []ast.Type{ast.TypeString},
        ReturnType: ast.TypeString,
        Callback: func(args []interface{}) (interface{}, error) {
            s := args[0].(string)
            h := md5.New()
            h.Write([]byte(s))
            hash := h.Sum(nil)
            encoded := base64.StdEncoding.EncodeToString(hash[:])
            return encoded, nil
        },
    }
}

Let me know your thoughts and if you believe this is the right way - if it is I can then submit a PR is TF core and a PR here which we can either wait to approve or just document that the setting won't work until the next release of core - not sure how you want to handle it.

[katbyte]

Hi @mb290,

After reviewing the issue and the comments in hashicorp/terraform#19245 and i agree with apparentlymart. This is a fairly unique to azure situation and as such should be implemented in the provider, not core. I suggest:

resource "azurerm_storage_blob" "testsb" {
...
  content_md5 = "${md5(file("text.txt"))}"
...

And then internally we can convert between hex and whatever format azure requires. As this is out preferred solution I am going to close hashicorp/terraform#19245 and #2209 as in it's current form cannot be merged.

[raidlman]

Any updates on this issue?

[tombuildsstuff]

👋

To give an update here: we're wrapping up switching over from the old Storage SDK to our new Storage SDK (Giovanni) which exposes the Content-MD5 headers here.

Whilst we've added some other requested features for Storage Blobs (including #2006 and #3876) - since this requires some thought/further investigation which we're not going to be able to fit in prior to 2.0 (which we're starting to focus on) - I'm going to move this into the Future milestone for the moment, where we should be able to take a look at this in time after 2.0.

Thanks!

[afdezl]

Any developments on this?

[slime-uk]

Good info here but looks like we need to wait until AzureRm provider 2.0 for this.

[matsest]

any news now that the provider is in 2.x?

[Humoiz]

I am working on this feature and will send out a PR tonight or tomorrow.

[hashibot[bot]]

This has been released in version 2.37.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.37.0"
}
# ... other configuration ...

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!