Azure-sdk-for-net: Long running operation failed with status 'Failed' when trying to create a resource using ARM templates

[rshirani]

I am trying to create a deployment based on https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-csharp-template/.

public static async Task<DeploymentExtended> CreateTemplateDeploymentAsync(
   TokenCredentials credential, 
   string groupName, 
   string deploymentName, 
   string subscriptionId)
   {
     Console.WriteLine("Creating the template deployment...");
     var templateParams = new Dictionary<string, Dictionary<string, object>>{
       { "adminPassword", new Dictionary<string, object>{{"value","Azure1234567"}}},
       { "adminUserName", new Dictionary<string, object>{{"value", "davidmu" } }}
     };

     var deployment = new Deployment();
     deployment.Properties = new DeploymentProperties
     {
       Mode = DeploymentMode.Incremental,
       Template = JObject.Parse(File.ReadAllText("Templates\\CreateVMTemplate2.json")),
       Parameters = templateParams
     };
     var resourceManagementClient = new ResourceManagementClient(credential) { SubscriptionId = subscriptionId };
     return await resourceManagementClient.Deployments.CreateOrUpdateAsync(groupName, deploymentName, deployment);
    }

I ran this multiple times and each time I get this exception. Is there anything that I am missing here?

Error: System.AggregateException: One or more errors occurred. ---> Microsoft.Rest.Azure.CloudException: Long running operation failed with status 'Failed'.
at Microsoft.Rest.Azure.AzureClientExtensions.d__12.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Rest.Azure.AzureClientExtensions.<GetPutOrPatchOperationResultAsync>d__01.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Management.ResourceManager.DeploymentsOperations.d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Management.ResourceManager.DeploymentsOperationsExtensions.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at ConsoleApplication1.Program.<CreateTemplateDeploymentAsync>d__3.MoveNext() in C:\Users\rostams\documents\visual studio 2015\Projects\ConsoleApplication1\ConsoleApplication1\Program.cs:line 151 --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task1.get_Result() at ConsoleApplication1.Program.Main(String[] args) in C:\Users\rostams\documents\visual studio 2015\Projects\ConsoleApplication1\ConsoleApplication1\Program.cs:line 49 ---> (Inner Exception #0) Microsoft.Rest.Azure.CloudException: Long running operation failed with status 'Failed'. at Microsoft.Rest.Azure.AzureClientExtensions.<GetPutOrPatchOperationResultAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Rest.Azure.AzureClientExtensions.d__01.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.ResourceManager.DeploymentsOperations.<CreateOrUpdateWithHttpMessagesAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.ResourceManager.DeploymentsOperationsExtensions.<CreateOrUpdateAsync>d__7.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at ConsoleApplication1.Program.d__3.MoveNext() in C:\Users\rostams\documents\visual studio 2015\Projects\ConsoleApplication1\ConsoleApplication1\Program.cs:line 151<---

[devigned]

@kirthik and @hovsepm we really should provide all HTTP first principles in exceptions. They should include the request / response headers, body, uri, etc., so that a consumer of the lib could reproduce the request using PS or curl. Beyond that, "Failed" is completely inadequate and should contain the information for why the deployment failed.

[jeffsaremi]

Do we have this now? Because I just got this error myself.

[troshko111]

Hit same error. Quick mitigation would be to open portal and look for the real error under resource group - deployments - your deployment

But the error is pretty ridiculous, it should include the proper amount of detail and honestly it's rather critical SDK bug, how come it's not fixed in half a year?

[markcowl]

CloudException already contains the entire request and response @devigned what is the actual ask here?

[devigned]

to actually show it in the exception output :)

[shahabhijeet]

@devigned could you clarify what you mean by exception output?
Do you think, we should provide an overload to exception.ToString()

[devigned]

@shahabhijeet the following is what it looks like when the Ruby SDK runs into a API request / response error. From the error output, you can see all of the details needed to understand what is happening at the HTTP level. You can see all HTTP request inputs, the HTTP outputs including body and headers, and you can see the configuration of the REST client. You have all information needed to debug the issue.

The output of the .NET error is just the Async stack trace. This provides zero information unless you have a debugger connected. The .NET error output above gives no information about the request or response. The default string representation of the error should provide assistance in determining the root cause of the error.

/var/lib/gems/2.3.0/gems/azure_mgmt_recovery_services_backup-0.8.0/lib/generated/azure_mgmt_recovery_services_backup/protection_containers.rb:221:in `block in list_async': { (MsRestAzure::AzureOperationError)
  "message": "MsRestAzure::AzureOperationError: CloudInternalError: Microsoft Azure Backup encountered an internal error.\r\nWait for a few minutes and then try the operation again. If the issue persists, please contact Microsoft support.",
  "request": {
    "base_uri": "https://management.azure.com",
    "path_template": "/Subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.RecoveryServices/vaults/{vaultName}/backupProtectionContainers",
    "method": "get",
    "path_params": {
      "vaultName": "TestVault",
      "resourceGroupName": "TestRG-RV",
      "subscriptionId": "########-####-####-####-############"
    },
    "skip_encoding_path_params": null,
    "query_params": {
      "api-version": "2016-06-01",
      "$filter": null
    },
    "skip_encoding_query_params": null,
    "headers": {
      "Content-Type": "application/json;charset=utf-8",
      "accept-language": "en-US",
      "x-ms-client-request-id": "3463914c-34bc-45d4-ba77-65852d8d8270"
    },
    "body": null,
    "middlewares": [
      [
        "MsRest::RetryPolicyMiddleware",
        {
          "times": 3,
          "retry": 0.02
        }
      ],
      [
        "cookie_jar"
      ]
    ],
    "log": null
  },
  "response": {
    "body": "{\"error\":{\"code\":\"CloudInternalError\",\"message\":\"Microsoft Azure Backup encountered an internal error.\\r\\nWait for a few minutes and then try the operation again. If the issue persists, please contact Microsoft support.\",\"target\":null,\"details\":null,\"innerError\":null}}",
    "headers": {
      "cache-control": "no-cache",
      "pragma": "no-cache",
      "content-length": "269",
      "content-type": "application/json",
      "x-ms-request-id": "4d55e5fe-331a-4f45-a6f7-3c5819b5c214",
      "x-ms-client-request-id": "3463914c-34bc-45d4-ba77-65852d8d8270, 3463914c-34bc-45d4-ba77-65852d8d8270",
      "strict-transport-security": "max-age=31536000; includeSubDomains",
      "server": "Microsoft-IIS/8.0",
      "x-powered-by": "ASP.NET",
      "x-ms-failure-cause": "service",
      "x-ms-ratelimit-remaining-subscription-reads": "14966",
      "x-ms-correlation-request-id": "4d55e5fe-331a-4f45-a6f7-3c5819b5c214",
      "x-ms-routing-request-id": "WESTEUROPE:20170109T105903Z:4d55e5fe-331a-4f45-a6f7-3c5819b5c214",
      "date": "Mon, 09 Jan 2017 10:59:03 GMT",
      "connection": "close"
    },
    "status": 500
  }
}

[shahabhijeet]

Assigning it to @markcowl. As per triage, @markcowl needs to talk to @devigned and get more information.

[markcowl]

Removing milestone info for triage

[yatesy]

Hi @markcowl. I get the same thing when trying to get a new sql server....

The message I get in powershell (which is totally useless) is...

New-AzureRMSQLServer : Long running operation failed with status 'Failed'.

When I go into the portal I get...

  "properties": {
        "statusMessage": "{\"status\":\"Failed\",\"error\":{\"code\":\"ResourceOperationFailure\",\"message\":\"The resource operation completed with terminal provisioning state 'Failed'.\",\"details\":[{\"code\":\"RegionDoesNotAllowProvisioning\",\"message\":\"Location 'Australia East' is not accepting creation of new Windows Azure SQL Database servers at this time.\"}]}}",

If the powershell task could present that back to me the detail which I can get out of the portal it would be much more useful.

[4c74356b41]

just ran into this, this is a bit sad.

[rokobasilisk]

I get the same error when submitting with Microsoft.Azure.Management.Fluent Deploy C# library to a web service.
Funnily enough it was working before the weekend, then got to work and no longer working.

I also get a equally vague error in the Activity Log in the portal.

[imaculate]

Hitting the same issue intermittently when using .NET SDK to create Network Security Group.

doing something like this
networkingClient.NetworkInterfaces.CreateOrUpdateAsync(...

[se7enack]

is this going to be fixed soon??

[openzig]

I have a solution here.
Catch CloudException and the error is contained in e.Response.Content

try {
await resourceClient.Deployments.CreateOrUpdateAsync(resourceGroup, deploymentName, deployment);
}
catch(CloudException e) {
Console.WriteLine(e.Response.Content)
}

[AlexGhiondea]

@rshirani is this still an issue you are facing?

[msftbot[bot]]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!