Google-cloud-dotnet: ERROR: (gcloud.app.deploy) Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation projects/

[Shanmt]

I am not able to deploy even the Google's .NetCore demo Hello World application to my App Engine in asia-south region. It is showing error as

ERROR: (gcloud.app.deploy) Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation projects/ ***/*/*

while deploying. I normally use Visual Studio + Google Cloud tool for deploying. I tried using gcloud cli also. But the issue persist. When try hitting server with the url, it is showing

"The server encountered a temporary error and could not complete your request. Please try again after 30 sec."

It was working fine till 21-April which I deploy at last. Yesterday when I try deploying first it prevented with BACKEND_SERVICES quota excceds. Updated Quota. Then trying deploying, I am seeing this issue. Able to see version as deployed, but no instance is getting created.

1) Tried Creating a new App Engine in another region and deployed the code, but iit is showing as Internal Error Occured.
2) Tried Deploying from latest gCloud Cli tool
3) Tried to deploy Hello World app in the GCP doc
4) Checked all the communities. Somebody opens a thread in SO, but not answered

Environment details

• OS: Windows 10
• .NET Core version: 2.1/ 2.2
• Package name and version:
  Google Cloud for Visual Studio 2.0.4
  Google Cloud SDK 290.0.0
  bq 2.0.56
  core 2020.04.21
  gsutil 4.49

Attaching Verbose Logs

Updating service [default] (this may take several minutes)...|DEBUG: Operation [apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067] not complete. Waiting to retry.
Updating service [default] (this may take several minutes)...|DEBUG: Operation [apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067] not complete. Waiting to retry.
Updating service [default] (this may take several minutes)...\DEBUG: Operation [apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067] not complete. Waiting to retry.
Updating service [default] (this may take several minutes)...\DEBUG: Operation [apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067] not complete. Waiting to retry.
Updating service [default] (this may take several minutes)...-DEBUG: Operation [apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067] complete. Result: {
    "done": true,
    "error": {
        "code": 4,
        "message": "App Engine Flexible timed out while configuring resources, internal operation projects/**projectid**/regions/asia-south1/operations/700238de-3add-4281-80f8-6dc66948a715"
    },
    "metadata": {
        "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
        "createVersionMetadata": {
            "cloudBuildId": "38b57f34-3b23-4484-9d69-0c581995947f"
        },
        "endTime": "2020-04-23T23:40:50.335Z",
        "insertTime": "2020-04-23T23:29:10.164Z",
        "method": "google.appengine.v1.Versions.CreateVersion",
        "target": "apps/**projectid**/services/default/versions/20200424t045856",
        "user": "[email protected]"
    },
    "name": "apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067"
}
Updating service [default] (this may take several minutes)...failed.
DEBUG: (gcloud.app.deploy) Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation projects/**projectid**/regions/asia-south1/operations/700238de-3add-4281-80f8-6dc66948a715
Traceback (most recent call last):
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\calliope\cli.py", line 983, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\calliope\backend.py", line 807, in Run
    resources = command_instance.Run(args)
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\surface\app\deploy.py", line 111, in Run
    return deploy_util.RunDeploy(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 634, in RunDeploy
    deployer.Deploy(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 431, in Deploy
    self.api_client.DeployService(new_version.service, new_version.id,
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\appengine_api_client.py", line 204, in DeployService
    return operations_util.WaitForOperation(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\operations_util.py", line 307, in WaitForOperation
    completed_operation = waiter.WaitFor(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 261, in WaitFor
    operation = PollUntilDone(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 322, in PollUntilDone
    operation = retryer.RetryOnResult(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\core\util\retry.py", line 229, in RetryOnResult
    if not should_retry(result, state):
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 320, in _IsNotDone
    return not poller.IsDone(operation)
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\operations_util.py", line 182, in IsDone
    raise OperationError(requests.ExtractErrorMessage(
googlecloudsdk.api_lib.app.operations_util.OperationError: Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation projects/**projectid**/regions/asia-south1/operations/700238de-3add-4281-80f8-6dc66948a715
ERROR: (gcloud.app.deploy) Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation projects/**projectid**/regions/asia-south1/operations/700238de-3add-4281-80f8-6dc66948a715

[jskeet]

Thanks for the report. I'm not very familiar with AppEngine flex, but I'll see what I can find out.

[jskeet]

Hmm... I've just tried to reproduce this, using exactly the "Hello World" sample in dotnet-docs-samples, and it worked for me.

Are you able to try this again? I wonder whether it was a transient error.

[Shanmt]

@jskeet, Really Appreciate your effort. Still I am getting the same error while deploying. I am still able to see msg as "The server encountered a temporary error and could not complete your request." while hitting this server url directly.

But now I am able to deploy the changes in a GAE Project running US East, and is working fine for now.

[jskeet]

But now I am able to deploy the changes in a GAE Project running US East, and is working fine for now.

Ah, so you're only seeing the problem in the asia-south region? I'll see if I can reproduce that next week - although it'll probably involve creating a new project...

[Shanmt]

I can share the Project details privately If you want. I am really sorrry to say that it will be difficult in my position to give you the access.

[jskeet]

If at all possible, it'll be best to do this without access to your project :) If it's only in a single region, that may well be something that's transient, too... if you're able to try again on Monday too, that would be helpful. But otherwise I should be able to create a project - it'll just take a little longer.

[jskeet]

Hi @Shanmt,

Before I start creating a project in asia-south, have you been able to check whether this is still an issue?

[Shanmt]

I tried again on the same project. It is still showing timeout issue while deploying. Showing 502 while hitting this server url directly. But the project that I created in region US-East is working fine as of now. I don't know if it is a particular issue with this app engine/ region wise server issue. Adding to the current issue, I am having the problem in Project X(asia-south) and Project Y(US east) is working fine . If Iocally init the gcloud with Project X (which is having 502 problem) and if I try to deploy Project Y from Visual Studio Google Cloud tools, even selected correct Project in the tools, it is failing to deploy.

[jskeet]

Okay, thanks. I'll create a project with AppEngine in asia-south and see if I can reproduce the problem.

[jskeet]

I've somewhat reproduced this problem, but with a different error message. Mine ended with just:

Updating service [default] (this may take several minutes)...failed.
ERROR: (gcloud.app.deploy) Error Response: [4] Timed out waiting for the app infrastructure to become healthy.

That may be because I didn't ask for verbose logs.

Trying with Python now.

[jskeet]

Just received the exact same issue for Python, so this is at least not .NET specific. Will chase it internally.

[Shanmt]

@jskeet , Thanks man, I am not sure, but the issue you are facing is probably due to some quota issues. https://stackoverflow.com/a/49385213/1657660. I think, I am facing Error 13 issue.

[jskeet]

Hmm... will check, thanks. This is a brand new project, so I'd hope that wasn't the problem. Interestingly, despite the timeout error, it has managed to create the version of the .NET application.

[Shanmt]

yes same here, But no instance is created. Right ?

[jskeet]

Yes. I'm running it again with --verbosity=debug. The Cloud Build part finished okay. No sign of quota issues yet.
(I've pinged a Product Manager internally, as he's likely to have a better idea of what's going on - I'm not an AppEngine specialist by any means.)

[jskeet]

Hmmm... that finished with a different verbose error to yours:

Result: {
    "done": true,
    "error": {
        "code": 4,
        "message": "Timed out waiting for the app infrastructure to become healthy."
    },
    "metadata": {
        "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
        "insertTime": "2020-04-28T09:40:41.554Z",
        "method": "google.appengine.v1.Versions.CreateVersion",
        "target": "apps/[...]/services/default/versions/20200428t103934",
        "user": "[...]"
    },
    "name": "apps/[...]/operations/26277863-31d5-4ae6-82df-02ab463afbe7"
}

[jskeet]

(Just to keep you informed, I'm talking about this with folks internally. I haven't forgotten about it!)

[Shanmt]

Thanks Jon Skeet..

[jskeet]

I'm afraid I can't reveal too much of the inner workings, but we understand why it's failing (and it looks like it is still failing).

The error message we should be getting is this:

The region asia-south1 does not have enough resources available to fulfill the request. Please try again later.

That would at least be slightly clearer than the timeout. I'm afraid I don't have any information as to when more resources will be available in that region.

At this point, I think we're at the limit of how I can help you. Are you happy for me to close this issue?

[jskeet]

(I didn't actually mean to close it until you'd confirmed you were happy for me to do so... I hit the wrong button. But I'll leave it that way for now...)

[Shanmt]

I am already running this on region Us East. I really appreciate what you are doing and very much thanks for your time. But other than that I've logged this issue on GCP bugsilla. I am not getting any response yet. I don't know what to say.

[jskeet]

I think everything's basically understood now:

• The asia-south1 region is apparently short of resources right now, at least in some respect. (There are complexities there I don't fully understand.)
• The deployment should fail quickly with an error message as per my earlier comment; we have an internal bug open for that.

I don't think you need to chase this any further on the GCP bug tracking - I think everything is covered internally now. I'm sorry that there's nothing we can do to make more resources available, but that's a somewhat different matter.

[rahulkadam]

HI @jskeet ,
are south asia region facing same issue, still ?

[jskeet]

@rahulkadam: I don't know whether it's "still" or "again" - but if you're seeing the same results, I expect it's due to the same cause.

[rahulkadam]

@jskeet thanks for quick reply,
can you provide some info from below error, that will be helpful

GCLOUD: ERROR: (gcloud.app.deploy) Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation projects/sattafantasy/regions/asia-south1/operations/cde8fe7f-5a57-40be-a49d-269cb2ed5cde

[jskeet]

@rahulkadam: I don't have any more information I can give, I'm afraid. I strongly suspect you'll see exactly the same error if you try deploying a Java or Python application; it's an App Engine issue rather than anything to do with .NET, and I really only know about the .NET client libraries.

You can try using --verbosity=debug as I did earlier in this thread and see what the result is, but I'm afraid I wouldn't have any more information for you. You'd be better off asking on one of the App Engine-specific support areas where the readers are likely to have more App Engine knowledge.

[rahulkadam]

thanks @jskeet , appreciate your help