Azure-sdk-for-net: Container.CreateIfNotExist throws exception if container exists

[dei79]

We are trying to create a container reference with the new 1.7.1 SDK but when the container exists it will throw an exception:

 //Get a reference to the container for which shared access signature will be created.  
 CloudBlobContainer container = blobClient.GetContainerReference(containerName);

 // create the container if not exists
 container.CreateIfNotExist();

This is the exception:

Microsoft.WindowsAzure.StorageClient.StorageClientException was unhandled by user code
  Message=The specified container already exists.
  Source=Microsoft.WindowsAzure.StorageClient
  StackTrace:
       at Microsoft.WindowsAzure.StorageClient.EventHelper.ProcessWebResponse(WebRequest req, IAsyncResult asyncResult, EventHandler`1 handler, Object sender) in C:\Users\deisenberg\azure-sdk-for-net\microsoft-azure-api\StorageClient\EventHelper.cs:line 84
       at Microsoft.WindowsAzure.StorageClient.CloudBlobClient.EndGetResponse(IAsyncResult asyncresult, WebRequest req) in C:\Users\deisenberg\azure-sdk-for-net\microsoft-azure-api\StorageClient\CloudBlobClient.cs:line 819
       at Microsoft.WindowsAzure.StorageClient.Tasks.WebRequestExtensions.<>c__DisplayClass1.<GetResponseAsync>b__0(IAsyncResult asyncresult) in C:\Users\deisenberg\azure-sdk-for-net\microsoft-azure-api\StorageClient\Tasks\WebRequestExtensions.cs:line 103
       at Microsoft.WindowsAzure.StorageClient.Tasks.APMTask`1.OnEnd(IAsyncResult ar) in C:\Users\deisenberg\azure-sdk-for-net\microsoft-azure-api\StorageClient\Tasks\APMTask.cs:line 162
  InnerException: System.Net.WebException
       Message=The remote server returned an error: (409) Conflict.
       Source=System
       StackTrace:
            at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
            at Microsoft.WindowsAzure.StorageClient.EventHelper.ProcessWebResponse(WebRequest req, IAsyncResult asyncResult, EventHandler`1 handler, Object sender) in C:\Users\deisenberg\azure-sdk-for-net\microsoft-azure-api\StorageClient\EventHelper.cs:line 77
       InnerException: 

[joeg]

We would have to look at a specific request, In rare cases you can get a conflict if the container is in the process of being deleted.

[dei79]

So is this issues solved or do you have any problems reproducing it?

[ghost]

As joeg mentioned above, you can still get a conflict if a container is currently being deleted, because the API can return neither true given that it could not create the container nor false because the container will be deleted soon. You can especially hit this behavior if you delete a container and try to recreate it again immediately.

If you would like us to investigate a specific request, please let us know.

[dei79]

hm, this crash happened with every container we are creating new with our tool. No pending remove operation or anything else was running. I will try it again with our tool.

[nstuke]

According to http://social.msdn.microsoft.com/Forums/windowsazure/en-US/8480223a-8cc7-4054-919c-a32ff02d5143/createifnotexists-concurrency-safe?forum=windowsazuredata, it looks like there is an issue with CreateIfNotExists:

"Storage team confirms that CreateTableIfNotExist and DeleteTableIfExist are broken in this way. They'll file a bug if one doesn't already exist.

In the meantime, a good workaround is to just call CreateTable and catch the StorageClientException check for this condition:

StatusCode == HttpStatusCode.Conflict && ExtendedErrorInformation.ErrorCode == TableErrorCodeStrings.TableAlreadyExists"

[sapiens]

2.5 years later and they still have this bug. Just wow.....

[abatishchev]

+1

[kerrpeter]

Why is this closed if its still a bug?

[btull89]

I would like to know why this issue was closed. I'm having the same issue with calling CreateIfNotExists within multiple hangfire jobs.
I'm never deleting the blob container..

[timosnel]

Still an issue here as well, I'm accessing a container in parallel and also receive the 409 conflict response. This should not be closed.

[michaeldaw]

I can confirm this error occurs. Please re-open.

[houseofcat]

I can confirm this is still an issue.

[rasmusjuhl]

I am currently having this issue.

[petermauger]

I have this problem when I try and connect to my private storage container from an external (not Azure based) development computer. When I run the same code on Azure it works (obviously). Just in case anyone else runs into this...

[codecadwallader]

I am also having this issue via the .NET SDK (v.8.4)

Workaround:

            // Explicitly checking for the container's existence before calling CreateIfNotExists is a workaround for a known SDK issue.
            if (!container.Exists())
            {
                container.CreateIfNotExists();
            }

[erezvani1529]

This behavior is expected and mentioned in Azure Storage Client changelog (effective for v8.0+) .
Please refer to the discussion on this issue on Azure Storage GitHub repo as well for more information.

Thanks,
Elham

[codecadwallader]

Thank you for following up. It's counter-intuitive to me that the method is designed to usually throw exceptions, but the workaround I mentioned above has been working fine.

[codecadwallader]

Scroll up a little to https://github.com/Azure/azure-sdk-for-net/issues/109#issuecomment-333512454

[abatishchev]

Missed that in the email thread somehow. Thanks!

public async Task TrueCreatIfNotExistsAsync(this Container container)
{
    if (await !container.ExistsASync())
    {
        await container.CreateIfNotExistsAsync();
    }
}

:(

[erezvani1529]

@codecadwallader ,

To make sure everyone is aware of the details: the approach you have suggested involves an unnecessary extra call to Storage Service which is undesirable in most cases. Currently the 409 conflict error is internally handled by SDK if the container already exists and is more performant since the need for an extra head call is removed.

If based on your use-case the majority of the calls to CreateIfNotExists will be against existing containers, I would suggest moving to a single step initialization on your containers.

(Please note that previously, AppInsight would report a failure if the container was created successfully which is now corrected to 409 in case the creation did not go through.)

Thanks!

[codecadwallader]

Deleted my first response as I was mistaken. You're right that I was seeing the 409 Conflict errors in Application Insights vs. unhandled exceptions. I will add the caches as you recommend.

[sadjadbp]

@erezvani1529

If majority of call is against existing container (we are calling CreateIfNotExists just to make sure it's created) should we be using

if (!container.Exists()) { container.CreateIfNotExists(); }

It would be just one head against container or should we still always call that CreateIfNotExists?

[BrainSlugs83]

It's ridiculous that there's a method named CreateIfNotExists that is supposed to throw if the container already exists. -- That's what the regular Create should do. -- additionally, the exception that comes back has the IsRetryable boolean set to true -- even though, no matter how many retries you do, it will never succeed. :(

[tynorton]

This is happening to us as well. This issue shouldn't be closed.

[andyblack19]

These 409 Conflict errors are considered failed dependencies in the 'Application Map' in App Insights. Also in the 'End-to-end transaction details' screen.
👎

[mathiash98]

Got this message today using <PackageReference Include="WindowsAzure.Storage" Version="9.3.0" /> dotnet core 2.1

The suggested code fix works:
if (!await container.ExistsAsync()) { await container.CreateIfNotExistsAsync(); }

[deyanp]

I can confirm this is still the case, as @andyblack19 reported 6 months ago:

These 409 Conflict errors are considered failed dependencies in the 'Application Map' in App Insights. Also in the 'End-to-end transaction details' screen.

And I don't see ExistsAsAsync method in Azure.Storage.Blobs 12.1.0 ...

Why can't we have a simple way of creating containers without throwing exception or generating error logs in App Insights, without the possibility to suppress them?

Pls reopen ..

[ghost]

I can confirm this is still the case, as @andyblack19 reported 6 months ago:

These 409 Conflict errors are considered failed dependencies in the 'Application Map' in App Insights. Also in the 'End-to-end transaction details' screen.

And I don't see ExistsAsAsync method in Azure.Storage.Blobs 12.1.0 ...

Why can't we have a simple way of creating containers without throwing exception or generating error logs in App Insights, without the possibility to suppress them?

Pls reopen ..

Agreed. Having 2.39k entries of 409 from blob storage in App Insights is unnecessary.

Re-open please.

[ErikV88]

Please re-open, i see the same problem. If I delete container from portal, while running function. But before CreateIfNotExistsAsync()
Or is it a way to update state?

[stofte]

As stated 8 years ago, we're seeing this exception with containers that exists and calling CreateIfNotExistsAsync.

Azure.RequestFailedException: 'The specified container already exists.
RequestId:783c97cc-801e-000a-2e53-f833c9000000
Time:2020-03-12T09:51:26.9840342Z
Status: 409 (The specified container already exists.)
ErrorCode: ContainerAlreadyExists

Headers:
Server: Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id: 783c97cc-801e-000a-2e53-f833c9000000
x-ms-client-request-id: f64121a5-8a5d-4674-a6ec-d4d4ee0e6312
x-ms-version: 2019-07-07
x-ms-error-code: ContainerAlreadyExists
Date: Thu, 12 Mar 2020 09:51:26 GMT
Content-Length: 230
Content-Type: application/xml
'

Using the latest (AFAIK, storage nuget packages seem as prolific as npm packages these days) Azure.Storage.Blobs version 12.4.0

[deyanp]

I have the feeling nobody from MS is noticing this issue as it is old and already closed ...

[stofte]

@deyanp #9758 seems to be the exact same bug

[RaymondJamin]

App insights log the 409 as an error. It just clutters up the error log now.

[BC89]

Can confirm that AppInsights logs it as error too and it's a bit of hassle to weed it out the logs