Azure-sdk-for-python: TLS error when using HTTPS/TLS URL (not permitted for non-TLS protected (non-https) URLs.)

[nofunatall]

Ubuntu 19.10
Linux rio-t460p 5.3.0-26-generic #28-Ubuntu SMP Wed Dec 18 05:37:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Python 3.7.5
azure-core==1.2.0
azure-identity==1.2.0
azure-storage-blob==12.1.0

Describe the bug
When trying to do a batch delete blob using a blob client created using a HTTPS URL the code errors with:

"Bearer token authentication is not permitted for non-TLS protected (non-https) URLs."
azure.core.exceptions.ServiceRequestError: Bearer token authentication is not permitted for non-TLS protected (non-https) URLs.

This same blob client connection can successfully be used to list blobs in a container.
The credentials used to create the client are service principal with rbac (client/secret/tenant) and the role is set to "Storage Blob Data Owner".

Steps to reproduce
Here is some example code to reproduce the error:

STORAGE_URL = 'https://' + STORAGE_ACCOUNT + '.blob.core.windows.net'
client_credentials = ClientSecretCredential(client_id=AZURE_CLIENT, client_secret=AZURE_SECRET, tenant_id=AZURE_TENANT)
blob_service_client = BlobServiceClient(account_url=STORAGE_URL, credential=client_credentials)
container_client = blob_service_client.get_container_client(STORAGE_CONTAINER)
def run_azure_blob_list():
  result = []
  blob_list = container_client.list_blobs(name_starts_with=STORAGE_FOLDER)
  for blob in blob_list:
    if blob.last_modified < IGNORE_DATE: 
      result.append([blob.name, blob.last_modified])
  return result
blob_list = run_azure_blob_list()
container_client.delete_blobs(*blob_list)

Container endpoint:
print(container_client.primary_endpoint)

https://REDACTED.blob.core.windows.net/uploads

[annatisch]

Thanks for the report @nofunatall - I will investigate this today :)

[annatisch]

Sorry for the delay @nofunatall!
Adding @xiafu-msft from the storage team for visibility.

[annatisch]

Update:
I have managed to repro this locally and have confirmed that it's actually an incompatibility in the forming the Storage "batch" requests, and the way oauth policy in azure-core enforcing the 'https' URL scheme.
This fix will need to be added to Azure core, so updating the labels.
Adding @lmazuel and @chlowell
The issue is that each part in the multi-part requests for storage do not use a fully qualified URL, instead it's just a path stub, e.g. /my-container/my-blob.txt

Therefore, when we hit the TLS check in the BearerTokenCredentialsPolicy the request is deemed invalid.

Details on the Batch request formatting are here:
https://docs.microsoft.com/en-us/rest/api/storageservices/blob-batch
Note that each "subrequest" is authenticated independently - and each subrequest is fed through the pipeline independently in order to generate those auth headers.

[lmazuel]

I suggest we don't apply this check when the URL is partial (parse and check the scheme, instead of startswith)

[annatisch]

@lmazuel, @xiangyan99 - did we land on a solution for this issue?

[lmazuel]

Yes, waiting for https://github.com/Azure/azure-sdk-for-python/pull/9821 to make it to azure-core (we missed the 1.3 timeline)

[annatisch]

This fix has been merged into master and will ship with the next release :)

[Petermarcu]

Should we close this issue now?

[annatisch]

Yes :)

[vsalvino]

Is this fix going out soon (1.4.*) - or is it not yet known when this will be released? I am still experiencing it in azure-core==1.4.0 and azure-storage-blobs==12.3.0.

[annatisch]

Thanks @vsalvino - I had forgotten that the blobs update hadn't actually been released yet. Re-opening for now.
@xiafu-msft - do you have an idea as to when the next blobs release will be?

[lmazuel]

Released part of 12.3.1 (see tags of this commit)