Azure-sdk-for-python: azure-storage-blob - sas credentials break client accessing local Azurite container

[kuba-lilz]

• Package Name: azure.storage.blob
• Package Version: 12.3.1
• Operating System: OSX 10.15.5
• Python Version: 3.5.6

Describe the bug
When using sas token credentials for BlobServiceClient, requests sent to Azurite container are malformed

To Reproduce
Start azurite container (e.g. arafato/azurite:2.6.5), then run following code that uses credentials and connection strings for azurite running locally:

shared_access_token = azure.storage.blob.generate_account_sas(
    account_name='devstoreaccount1',
    account_key="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==",
    resource_types=azure.storage.blob.ResourceTypes(service=True, container=True, object=True),
    permission=azure.storage.blob.AccountSasPermissions(read=True, list=True),
    expiry=datetime.datetime.utcnow() + datetime.timedelta(hours=100),
    start=datetime.datetime.utcnow() - datetime.timedelta(hours=100))

storage_client = azure.storage.blob.BlobServiceClient.from_connection_string(
    conn_str="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;TableEndpoint=http://localhost:10002/devstoreaccount1;QueueEndpoint=http://localhost:10001/devstoreaccount1;BlobEndpoint=http://localhost:10000/devstoreaccount1",
    credential=shared_access_token)

for container in storage_client.list_containers():
    print(container.name)

Expected behavior
Containers can be enumerated.

Additional context

When BlobServiceClient is created using credential=None, then valid requests are sent to Azurite and results returned.
Example of a valid request (from docker logs):

/devstoreaccount1/?comp=list 

When BlobServiceClient is created using credential=shared_access_token, then requests to Azurite contain two slashes after devstoreaccount1 part of url, instead of just one, e.g.:

/devstoreaccount1//?st=2020-06-06T00%3A13%3A14Z&ss=b&se=2020-06-14T08%3A13%3A14Z&comp=list&sv=2019-07-07&srt=sco&sp=rl&sig=Vkr7NDuOyGIrZaGeP1mTJcrLpgpl%2BbIWAb3hT93uZHk%3D 

which of course means Azurite can't understand the request.
No such problems happen when using real azure storage blob accounts in Azure.

[kuba-lilz]

Here is a complete script (that runs a small flask server) from which above code snipped was taken away in case that's any help:

import datetime

import azure.storage.blob
import flask

APP = flask.Flask("my_app")


@APP.route("/")
def index():

    shared_access_token = azure.storage.blob.generate_account_sas(
        account_name='devstoreaccount1',
        account_key="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==",
        resource_types=azure.storage.blob.ResourceTypes(service=True, container=True, object=True),
        permission=azure.storage.blob.AccountSasPermissions(read=True, list=True),
        expiry=datetime.datetime.utcnow() + datetime.timedelta(hours=100),
        start=datetime.datetime.utcnow() - datetime.timedelta(hours=100))

    print("shared access token is\n{}\n\n".format(shared_access_token))

    storage_client = azure.storage.blob.BlobServiceClient.from_connection_string(
        conn_str="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;TableEndpoint=http://localhost:10002/devstoreaccount1;QueueEndpoint=http://localhost:10001/devstoreaccount1;BlobEndpoint=http://localhost:10000/devstoreaccount1",
        credential=shared_access_token)

    container_client = storage_client.get_container_client("sketchpad")

    images_blobs = [blob for blob in container_client.list_blobs() if ".jpg" in blob.name]

    urls = []

    for image_blob in images_blobs:

        blob_client = container_client.get_blob_client(image_blob)
        urls.append('<a href="{}">blob here</a>'.format(blob_client.primary_endpoint))

    images_data = "<br>".join(urls)

    response = ("""
    <html>
    <body>
    {}
    </body>
    </html>
    """.format(images_data))

    return response


APP.run(use_reloader=True)

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[xiafu-msft]

Hi @kuba-lilz
Thanks for reporting this! We have logged a bug and it will be fixed later!
Hope it doesn't block you! Let me know if you have any concern

[hayesgb]

I'm seeing similar behavior with azure-storage-blob v12.4.0, which manifests in the dask/adlfs package as reported here.

After starting azurite locally using docker, execution of the following:

bbs = BlobServiceClient.from_connection_string(
    conn_str="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;TableEndpoint=http://localhost:10002/devstoreaccount1;QueueEndpoint=http://localhost:10001/devstoreaccount1;BlobEndpoint=http://localhost:10000/devstoreaccount1",
    key="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==")
bbs.create_container("data")
cons = bbs.list_containers()
print([c for c in cons])

returns the following error:

---------------------------------------------------------------------------
StorageErrorException                     Traceback (most recent call last)
~/anaconda3/envs/adlfs/lib/python3.7/site-packages/azure/storage/blob/_models.py in _get_next_cb(self, continuation_token)
    398                 cls=return_context_and_deserialized,
--> 399                 use_location=self.location_mode)
    400         except StorageErrorException as error:

~/anaconda3/envs/adlfs/lib/python3.7/site-packages/azure/storage/blob/_generated/operations/_service_operations.py in list_containers_segment(self, prefix, marker, maxresults, include, timeout, request_id, cls, **kwargs)
    335             map_error(status_code=response.status_code, response=response, error_map=error_map)
--> 336             raise models.StorageErrorException(response, self._deserialize)
    337 

StorageErrorException: Operation returned an invalid status 'Bad Request'

During handling of the above exception, another exception occurred:

HttpResponseError                         Traceback (most recent call last)
<ipython-input-3-beaff551c875> in <module>
      4 bbs.create_container("data")
      5 cons = bbs.list_containers()
----> 6 print([c for c in cons])

<ipython-input-3-beaff551c875> in <listcomp>(.0)
      4 bbs.create_container("data")
      5 cons = bbs.list_containers()
----> 6 print([c for c in cons])

~/anaconda3/envs/adlfs/lib/python3.7/site-packages/azure/core/paging.py in __next__(self)
    120         if self._page_iterator is None:
    121             self._page_iterator = itertools.chain.from_iterable(self.by_page())
--> 122         return next(self._page_iterator)
    123 
    124     next = __next__  # Python 2 compatibility.

~/anaconda3/envs/adlfs/lib/python3.7/site-packages/azure/core/paging.py in __next__(self)
     72             raise StopIteration("End of paging")
     73 
---> 74         self._response = self._get_next(self.continuation_token)
     75         self._did_a_call_already = True
     76 

~/anaconda3/envs/adlfs/lib/python3.7/site-packages/azure/storage/blob/_models.py in _get_next_cb(self, continuation_token)
    399                 use_location=self.location_mode)
    400         except StorageErrorException as error:
--> 401             process_storage_error(error)
    402 
    403     def _extract_data_cb(self, get_next_return):

~/anaconda3/envs/adlfs/lib/python3.7/site-packages/azure/storage/blob/_shared/response_handlers.py in process_storage_error(storage_error)
    145     error.error_code = error_code
    146     error.additional_info = additional_data
--> 147     raise error
    148 
    149 

HttpResponseError: Operation returned an invalid status 'Bad Request'
ErrorCode:None

Execution with azure-storage-blob 12.3.2 returns:

[{'name': 'data', 'last_modified': datetime.datetime(2020, 8, 15, 13, 31, 55, tzinfo=datetime.timezone.utc), 'etag': '"d-uncJpKbVsSBPdz3DP+1dHwJdN1k"', 'lease': {'status': 'unlocked', 'state': 'available', 'duration': None}, 'public_access': None, 'has_immutability_policy': False, 'has_legal_hold': False, 'metadata': {}, 'encryption_scope': None}]

[tasherif-msft]

Hi there,
The changes to fix this issue have been made and it will be available in our September release.
If this is an urgent issue or if there are any concerns, let me know!