Google-cloud-dotnet: Missing GOOGLE_APPLICATION_CREDENTIALS env variable in flex for .NET core?

[mastoj]

I'm trying to use datastore from a F# .NET Core 2.0 application. The code works fine locally and I expected it to do so on AppEngine as well.

Info about the app:

• Docker base image: gcr.io/google-appengine/aspnetcore:2.0
• Version of datastore package: 2.1.0-alpha00
• Code to create datastoredb: DatastoreDb.Create("MY GOOGLE PROJECT")
• App yaml

env: flex
runtime: custom
service: service-name
manual_scaling:
  instances: 1

Result:

Unhandled Exception: System.TypeInitializationException: The type initializer for '.$Program' threw an exception. ---> System.InvalidOperationException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
at Google.Api.Gax.TaskExtensions.WaitWithUnwrappedExceptions(Task task)
at Google.Api.Gax.Grpc.ChannelPool.GetChannel(ServiceEndpoint endpoint)
at Datastore.getToken(String kind, String name) in /Users/tomasjansson/git/unacast/CoreService/src/FatSlack/Datastore.fs:line 7

Expected result:

An instance of the DataStoreDb is created.

I thought that those where somehow automatically mounted when a custom flex application was started. I realize that this might not be an issue with the framework, but rather how custom flex appications are started on app engine.

[jskeet]

Hmm... I don't know much about custom flex images, although I would still expect us to detect that we're running on Compute Engine. (It should see the metadata server and detect it that way.)

@mastoj As an experiment, could you see what Google.Apis.Auth.OAuth2.IsRunningOnComputeEngine().Result returns? I suspect it's either not reaching the metadata server, or not getting the header it expects. Hmm.

@ivannaranjo, do you know anything about this? You know more about Flex than I do...

[mastoj]

@jskeet, strange, the result of Google.Apis.Auth.OAuth2.ComputeCredential.IsRunningOnComputeEngine().Result returns true.

I did some other change as well and now I'm getting a little bit longer which is strange. What I changed was using Suave.IO instead of ASP.NET. But I still doesn't get the result I expect, instead I get a null pointer where I shouldn't get one. I'll update with more information soon.

[jskeet]

Hmm... when it was failing under ASP.NET, did it do so reliably? (If you're really easily able to retest the previously-failing code, it would be great if you could do so - I wonder whether this was the result of a temporary network outage or similar. Don't worry if it's not simple to do though.)

It's probably worth reporting the NRE in a separate issue unless it looks like it's to do with credentials.

[jskeet]

(Marking as a bug until we know otherwise. If we are able to reproduce, we should fix it ASAP - if we can't, we may need to leave it, unfortunately.)

[mastoj]

I got it working with suave, but I didn't change any of the relevant the code when I did so. I would expect that I probably changed something else as well when making the change from ASP.NET to suave that made it work, because the actual code that failed where actually not called from ASP.NET.

Not that easy re re-produce to the exact state, since I hadn't the code in a git repo at the time. Sorry.

@jskeet, you can close if you want to since it was most likely a user error, which I somehow find without knowing.

Great response time on the issue, love it!

[jskeet]

Righto - will close for now, but please don't hesitate to add a comment if it happens again, and I'll reopen and investigate.

[mastoj]

@jskeet, there might be any issue somewhere. I had the app running, but then I stopped the service for a while and that caused the issue to come back.

So same app/binaries two different results.

[mastoj]

I don't have a way to reproduce it every single time so far.

[jskeet]

Okay, thanks for reporting. I'm on holiday this week, so don't have good network access for this sort of thing, but I'll have a look when I'm back next week. Any other environment information you can give would be handy.

[mastoj]

Let me know what other environment information you need?

The problem seems to reproducible for me when I do a redeploy of an app but use the same app engine version (the one I provide with --version to gcloud app deploy). If I delete the app engine version and then deploy with that version it will work.

[jskeet]

That's good to know. With the image information at the start, that should be enough...

[mastoj]

I can also confirm that deploying a new version, not overwriting an existing one, worked.

[jskeet]

Just to keep communications up to date, I haven't had time to reproduce this just yet, and probably won't for the next few days - lots of other stuff is going on, unfortunately. It's still very much on my radar though.

[jskeet]

Hmm. I've just tried this three times, deploying to version 1 each time, modifying the code in between.

It's the stock ASP.NET Core application, but with the contact details page calling GoogleCredential.GetApplicationDefault and reporting the result - it's worked every time. (Given your diagnosis, I'd expect it to work the first time but not the second or third.) It's C# rather than F#, but I'd be very surprised if that made a difference.

My Dockerfile is just

FROM gcr.io/google-appengine/aspnetcore:2.0
COPY . /app
WORKDIR /app
EXPOSE 8080
ENV ASPNETCORE_URLS=http://*:8080
ENTRYPOINT ["dotnet", "Issue1394.dll"]

I'm deploying with:

dotnet publish -c Release
gcloud beta app deploy bin\Release\netcoreapp2.0\publish\app.yaml --version=1

... can you spot any differences with what you're doing?

(And if you are still in a position where you can easily try to reproduce this, are you still observing it? If the problem has gone away, I can ask internally to see if there's anything that's changed in the last week and a half to explain it.)

[mastoj]

Can't spot any obvious differences. I'll try to see if I can't put together
another smaller example later this weekend or next early next week that I
can share with you.
On Fri, 1 Sep 2017 at 16:35, Jon Skeet [e-mail] wrote:

Hmm. I've just tried this three times, deploying to version 1 each time,
modifying the code in between.

It's the stock ASP.NET Core application, but with the contact details
page calling GoogleCredential.GetApplicationDefault and reporting the
result - it's worked every time. (Given your diagnosis, I'd expect it to
work the first time but not the second or third.) It's C# rather than F#,
but I'd be very surprised if that made a difference.

My Dockerfile is just

FROM gcr.io/google-appengine/aspnetcore:2.0
COPY . /app
WORKDIR /app
EXPOSE 8080
ENV ASPNETCORE_URLS=http://*:8080
ENTRYPOINT ["dotnet", "Issue1394.dll"]

I'm deploying with:

dotnet publish -c Release
gcloud beta app deploy binReleasenetcoreapp2.0publishapp.yaml --version=1

... can you spot any differences with what you're doing?

(And if you are still in a position where you can easily try to reproduce
this, are you still observing it? If the problem has gone away, I can ask
internally to see if there's anything that's changed in the last week and a
half to explain it.)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-dotnet/issues/1394#issuecomment-326597054,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAemsMf5mGPTV8m6cWdD2a3zzF3cOP_pks5seBYqgaJpZM4O9wgj
.

[jskeet]

Hi @mastoj - any luck reproducing this? I'd love to get to the bottom of it.

[mastoj]

Sorry for slow response here. Haven’t had the time to set up a smaller
example, due to too much work.

However, I did try to change the app that I had problem with to deploy to
the same version every time and now it works. Strange.
On Mon, 18 Sep 2017 at 17:51, Jon Skeet [e-mail] wrote:

Hi @mastoj https://github.com/mastoj - any luck reproducing this? I'd
love to get to the bottom of it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-dotnet/issues/1394#issuecomment-330266967,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAemsIaUJCpMKpRpLkfSOWv2gIy7nL-Xks5sjpGHgaJpZM4O9wgj
.

[jskeet]

Okay - thanks very much for trying again. Maybe it was something within GAE that's been fixed. I'll close this for now, but do please add another comment if you see this again, and I'll reopen it right away.

[mastoj]

Same problem again now @jskeet. Tried to redeploy my app with --version main and it failed exactly the same way as before. I'll try to get some time to put up a smaller sample that reproduces it.

[jskeet]

Great, that would be really helpful, thanks. I'll see if the solution I had before is reproducing it at the moment...

[mastoj]

I can't guarantee that I'll get to it this week though.

[jskeet]

Reopening - I'm still failing to reproduce though :(

[Jonathan34]

may be overwriting the version instance with flex is a bit messed up. We do deploy our .net core 2.0 apps in a flex environment but we specifically want new versions in case we want to rollback a service. Of course then, you have to clean up the versions... that could eventually be a rule for app engine to do automatically (prune instances that are stopped for 1 month).

I noticed @jskeet is using gcr.io/google-appengine/aspnetcore:2.0 as base image. I am using microsoft one microsoft/aspnetcore:2.0.0. Wondering if that could be a difference in the base image?

[jskeet]

@Jonathan34: I'm using that one because it's the one @mastoj is using. I'm trying to be as similar as I can, without venturing into F# ;)

[mastoj]

I've been stopping and restarting the app multiple times lately, and it works as expected. So I'm not able to reproduce this anymore.

Really unsure what's the cause of the issue.

Ok to close if you want to @jskeet. I'll re-open (again) if I see this again.

[jskeet]

I can see this being "one of those bugs" - will close for now, at which point no doubt it'll start happening again...

[helgejfo]

I have experienced this issue on two occasions now, once using a .NET Core 2.0 app and another time using a .NET Core 1.1 app, both times hosted on App Engine Flexible in different Google projects. I have not been able to create a good reproducer either. It usually goes away after a redeploy, but it has persisted on redeploys as well.
Exception:

System.InvalidOperationException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. 
at Google.Api.Gax.TaskExtensions.WaitWithUnwrappedExceptions(Task task) 
at Google.Api.Gax.Grpc.ChannelPool.GetChannel(ServiceEndpoint endpoint) 
at GCDataStore..ctor(String kind, String projectId) 
in d:\a\1\s\lib\..\GCDataStore.cs:line 19 

Fails on:

DatastoreDb.Create(projectId);

Docker configurations:

FROM microsoft/aspnetcore:1.1
ADD ./app/MyApp /app                                                                                                                                                                                           
ENV ASPNETCORE_URLS=http://*:${PORT}                                                                                                                                                                 
WORKDIR /app                                                                                                                                                                                           
ENTRYPOINT [ "dotnet", "MyApp.dll" ]

FROM microsoft/aspnetcore:2.0
ADD ./app/MyApp /app                                                                                                                                                                                           
ENV ASPNETCORE_URLS=http://*:${PORT}                                                                                                                                                                 
WORKDIR /app                                                                                                                                                                                           
ENTRYPOINT [ "dotnet", "MyApp.dll" ]



md5-06f08b0a8312f4393df1018a78e88be4



env: flex
runtime: custom
service: my-service

I realize this is hard to debug without a proper reproducer. Any suggestions on what I can do to get more information out of the running application or the GCP environment that can help?

I will try moving some of the applications to the aspnetcore runtime in app.yaml and see if the issue goes away over time.

[jskeet]

@helgejfo: More info is always welcome, thanks!

Things it would be useful to log:

• await Google.Apis.Auth.OAuth2.ComputeCredential.IsRunningOnComputeEngine()
• await Google.Api.Gax.Platform.InstanceAsync()

[helgejfo]

Thanks. Not seeing the issue at the moment, but will report back if I do. It currently reports:

True
[GAE: ProjectId=...]

[jskeet]

Right - both of those are what I'd expect in the success case. Are you able to leave that logging in place (somewhere earlier than the failure) so that if it comes back, we can see the difference?

(I'm really, really grateful for the help trying to diagnose this.)

[helgejfo]

Will do. Happy to help.

[jskeet]

I've now seen this on a VM in GCE - no AppEngine Flex involved or anything like that. Just running a .NET Core tool that failed. On the second attempt within the same VM, it worked. (Possibly importantly, this was the first thing that I'd run since rebooting the VM.)

Now that we know it can be seen on a regular VM, it may be easier to diagnose. I'll bash at it a bit more myself...

[mastoj]

“Happy” someone else hit this issue as well so it wasn’t just me
hallucinating:)

I’m not deploying as frequently at the moment, I guess that is why I don’t
see it.
On Thu, 30 Nov 2017 at 14:55, Jon Skeet [e-mail] wrote:

I've now seen this on a VM in GCE - no AppEngine Flex involved or anything
like that. Just running a .NET Core tool that failed. On the second attempt
within the same VM, it worked. (Possibly importantly, this was the first
thing that I'd run since rebooting the VM.)

Now that we know it can be seen on a regular VM, it may be easier to
diagnose. I'll bash at it a bit more myself...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-dotnet/issues/1394#issuecomment-348194293,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAemsMJnk27ilrf1QFHlR_0n77BT-n-Kks5s7rO2gaJpZM4O9wgj
.

[jskeet]

I've been trying to reproduce this, but to no avail despite repeated VM restarts etc. I haven't forgotten about it...

[helgejfo]

Good to know. I have not seen the issue since last time either.

[jskeet]

Now managed to reproduce it once with the adhoc test. Interestingly, it looks like we did get some information (middle line)

MakeLanguageRequest: Exception: InvalidOperationException (...)
ShowPlatform: [GCE: ProjectId='xxx', InstanceId='xxx', ZoneName='projects/xxx/zones/us-central1-c']
ShowRunningOnComputeEngine: False

(I've replaced the real values with xxx. But they looked okay.)

[mastoj]

Great to see progress on this weird bug.
On Fri, 8 Dec 2017 at 15:47, Jon Skeet [e-mail] wrote:

Now managed to reproduce it once with the adhoc test. Interestingly, it
looks like we did get some information (middle line)

MakeLanguageRequest: Exception: InvalidOperationException (...)
ShowPlatform: [GCE: ProjectId='xxx', InstanceId='xxx', ZoneName='projects/xxx/zones/us-central1-c']
ShowRunningOnComputeEngine: False

(I've replaced the real values with xxx.)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-dotnet/issues/1394#issuecomment-350280840,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAemsBQrDGW5d6ufmZ_yc9x40RdTsszyks5s-UwTgaJpZM4O9wgj
.

[jskeet]

So, I've added more diagnostics - basically making the HTTP request in my own code directly, and timing it.
Since making the change, I haven't seen the problem again - and more importantly, I haven't seen a request take more than a second (which is our timeout). I have seen it take about 0.65s though - suggesting that 1s isn't outside the bounds of possibility.

I wonder whether we should bump the timeout up to 2s...

[art-grig]

Hi all,
seems I've got exactly the same bug with .Net Core 2.0 on flex engine, it also dissapears after redeploying, but as far as redeploy takes ~8+ mins it's really painful.

My app.yaml:

env: flex
runtime: aspnetcore

[jskeet]

@art-grig: How reproducible is this for you? So far I've seen it once after trying about a hundred reboots on a VM - and I've never managed to reproduce it in an ASP.NET Core app. To be clear, I absolutely believe everyone reporting it - I'm just trying to find the most consistent way of reproducing this so we can look into fixing it. This is obviously a very real and annoying problem that needs fixing.

[art-grig]

Unfortunately, in my case it has appeared just twice last month and I've always fixed it with redeploying.
Not really sure what might be the root cause. Maybe something with versioning?
That's how I deploy it:

gcloud app deploy -v v018

[jskeet]

No, I don't think you're doing anything wrong at all. I'll have another go at reproducing this on Flex in the New Year. Maybe if I deploy requiring a minimum of 10 instances, and auto-detect the failure and log it, I'll be able to reproduce it more easily... although doing the logging after it's failed to get the default credentials could require a bit of work :)

[jskeet]

@mastoj @art-grig @helgejfo: Could you let me know which GCP regions you're running in?

I now have an automated system for stopping/starting AppEngine services every few minutes and gathering the results of their logging. Unfortunately that hasn't shown the problem yet, or the metadata server taking longer than a second to respond... the closest has been about 700ms.

Different data centers appear to give me different results though, so that may be part of reproducing the issue...

[art-grig]

@jskeet, I am running in us-central.

[helgejfo]

@jskeet, I have seen the issue in both us-central and europe-west.

[jskeet]

Thanks for the details folks. I'm running tests in us-central and europe-west. I still haven't seen anything above 700ms (after literally thousands of restarts), but this could be something that's specific to a DC at a particular time. I'll keep running the tests and see if anything shows up.

If anyone has any more insight - or purely speculation at this point! - please share. It's frustrating for all of us, not being able to reproduce and fix this :(

[mastoj]

us-central is what I'm using.

[jskeet]

Okay... I've now merged PRs to update the timeout to 2000ms for both credentials and platform detection. The timeout is handled slightly differently in different cases, so it's not entirely clear to me what's going to happen, but I don't think I want to change anything else right now. None of this will take effect until the next release of GAX and Google.Apis.Auth, of course - and even then you'd need to manually update the dependencies. I'll leave this bug open until those releases have gone out, but I'm not expecting to take further code action before the releases (unless happens to find a better way of reproducing the problems).

[MaikuMori]

Happened to us as well. us-central. Don't have any debug logs except the exception:

System.InvalidOperationException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. at Google.Apis.Auth.OAuth2.DefaultCredentialProvider.<CreateDefaultCredentialAsync>d__11.MoveNext()

[jskeet]

@chrisdunelm Maybe we should do a patch release of the auth library to get the increased timeout going? (I'm not sure it's worth waiting for the next minor release.)

[art-grig]

@jskeet I am just getting that error right now again, in case if you want to take a look

[jskeet]

@art-grig: Could you manually add a dependency on Google.Apis.Auth version 1.32.1? (You'll already have an implicit dependency on an earlier version.) That has the longer timeout. If you still see the problem with the longer timeout, I'll need to go back to the drawing board.

I completely understand if you can't easily change your dependencies. Any extra information you can give about your app would be very welcome though.

[vuukle]

@jskeet we are trying your suggestion and will let you know ASAP. What other information could I provide? Im happy to do a team viewer session or google hangout to help solve this issue

[jskeet]

@vuukle: That's awesome - thanks! In general, all of these are useful:

• Deployment environment (Container Engine, AppEngine Flex etc)
• Deployment image (some people use our image, others use a vanilla one)
• All package versions (both Google and not)
• GCP Region (us-central, europe-west etc)
• What you were doing - redeploying an existing version, deploying a new version etc
• Any stats around how many failures you're getting over what period of time and out of how many deployments.

In short, anything that can help establish a pattern...

[art-grig]

@jskeet Thanks for advice, it worked, but not sure if it's really about updating dependencies or just one more redeploying

[jskeet]

@art-grig: Understood. Unless it was already failing reliably, it's hard to be sure of anything :(

[art-grig]

@jskeet it's me again, and we faced this issue again :(

1. Appengine flex
2. Standard image
   

1. us-central
2. deploying new version (with gcloud tools for vs2017)
3. not sure about stats, but it happens quite often

[jskeet]

@art-grig: Okay, that's unfortunate but very helpful in terms of confirming that it still happens with 1.32.1.

Back to the drawing board... can I just check, am I right in saying that you see this immediately after deployment, i.e. the deployed container never makes any successful requests? If you're seeing it after a container has already been successfully active for a while, that would suggest an entirely different problem.

[art-grig]

@jskeet Actually it makes successful requests, furthermore it's blinking: like you are getting 200-200-500-200-500-500-200-500 statuses while trying to access any resource.

[jskeet]

@art-grig: From a single server? Okay, that's really interesting and very, very bizarre. Your initial report showed a TypeInitializationException - I assume that's not what you're getting in your "blinking" example? Can you provide a new stack trace if possible?

[jskeet]

Anyone else experiencing this issue - could you each clarify which of the following situations you're in, on a per server basis? (Multiple servers leading to an overall "sometimes working, sometimes doesn't" result isn't as interesting.)

1. Only "new" instances fail: if a server ever successfully services a request that needs an API, it keeps doing so until it's taken down
2. An instance starts failing at some point during its lifetime, having succeeded in making some API requests - but then fails all future requests
3. An instance fails to make some API requests but continues to succeed to make some too
4. Something else

@mastoj @vuukle @MaikuMori @helgejfo

[helgejfo]

In my case I have seen both 1 and 3.

Case 1 usually happens where the DatastoreDb is created in a singleton service, so naturally it will fail on startup without any retry / recovery logic.

Case 3 happens when DatastoreDb is created on a per request basis, however I cannot say if this is because one App Engine instance fails and another succeeds or if it both fails and succeeds on the same instance. Usually running two instances.

[jskeet]

@helgejfo: Thanks; I suspect you're right that case 3 is just a matter of having multiple instances. I'm going to try more tests bringing up and tearing down lots of instances, and creating actual DatastoreDb instances rather than just code that I'd expect to be equivalent. It'll be so much easier to fix if I can reproduce this, even if it's still somewhat transient.

[vuukle]

@jskeet is there a minute chance it could be related to the version number we promote e.g. v22 or v02022018t20189, I always deploy using Visual Studio instead of using powershell, next time i see this issue .. i am happy to do a teamviewer/hangout session if it helps at all

[jskeet]

@vuukle: I can't see how it could be, but I'd be happy to be contradicted if anyone spots a pattern. My own tests have been initially deployed from VS as well, so have that style of version number. I'm not immediately sure what I'd be able to see on a Hangout, but I appreciate the offer and will think about it further. A complete stack trace would be really helpful if you have one.

[jskeet]

@vuukle: Could you look at the logs to tell whether the successful requests are going to a different instance?

[jskeet]

Not at my computer right now, but will check when I'm back.

[jskeet]

Right, I'm back now. Those requests are definitely to different instances - the appengine.googleapis.com/instance_name part confirms that.

For the moment, until we have concrete evidence otherwise, I'm going to assume that if an instance is broken, it's broken right from the first request, and stays broken. Today I'm going to d more work trying to reproduce this. We'll see what happens...

[vuukle]

Great, i deleted my comments if you need debug information i am available on [email protected]

I am doing a redeploy and deleting all old instances since we only use one flex instance

[bgrainger]

I think I'm also affected by this problem (i.e., not user error but I'm happy to be proven wrong). I just added Stackdriver error reporting to a simple C# ASP.NET Core application hosted in App Engine flexible environment (us-central) as per these instructions. I thought it would "just work" as per this information:

The Google.Cloud.Diagnostics.AspNetCore client library should work without the need to manually provide authentication credentials for instances running on Google Cloud Platform, as long as the Stackdriver Trace API access scope is enabled on that instance.

App Engine

On Google App Engine flexible environment, the Stackdriver Trace API access scope is enabled by default, and the Google.Cloud.Diagnostics.AspNetCore client library can be used without needing to provide credentials or a project ID.

When I run gcloud app deploy --version v9 I get:

Services to deploy:

descriptor:      [XXX\bin\Release\netcoreapp2.0\publish\app.yaml]
source:          [XXX\bin\Release\netcoreapp2.0\publish]
target project:  [XXX]
target service:  [default]
target version:  [v9]
target url:      [https://XXX.appspot.com]


Do you want to continue (Y/n)?

Beginning deployment of service [default]...
Building and pushing image for service [default]
Started cloud build [f8bd894d-54f4-48bf-9f85-fe79e087925c].
To see logs in the Cloud Console: https://console.cloud.google.com/gcr/builds/f8bd894d-54f4-48bf-9f85-fe79e087925c?project=XXX
------------------------------------------------------------------------------------ REMOTE BUILD OUTPUT -------------------------------------------------------------------------------------

starting build "f8bd894d-54f4-48bf-9f85-fe79e087925c"

[ snip ]

Updating service [default] (this may take several minutes)...failed.
ERROR: (gcloud.app.deploy) Error Response: [9]
Application startup error:
Application startup exception: System.InvalidOperationException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
   at Google.Api.Gax.TaskExtensions.WaitWithUnwrappedExceptions(Task task)
   at Google.Api.Gax.Grpc.ChannelPool.GetChannel(ServiceEndpoint endpoint)
   at Google.Cloud.Diagnostics.Common.EventTarget.ForLogging(LogTarget logTarget, String logName, LoggingServiceV2Client loggingClient, MonitoredResource monitoredResource)
   at Google.Cloud.Diagnostics.Common.ErrorReportingContextExceptionLogger.Create(String projectId, String serviceName, String version, ErrorReportingOptions options)
   at Google.Cloud.Diagnostics.AspNetCore.ErrorReportingExceptionLoggerExtension.AddGoogleExceptionLogging(IServiceCollection services, Action`1 setupAction)
   at XXX.Startup.ConfigureServices(IServiceCollection services) in XXX\Startup.cs:line 35
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
crit: Microsoft.AspNetCore.Hosting.Internal.WebHost[6]
      Application startup exception
System.InvalidOperationException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
   at Google.Api.Gax.TaskExtensions.WaitWithUnwrappedExceptions(Task task)
   at Google.Api.Gax.Grpc.ChannelPool.GetChannel(ServiceEndpoint endpoint)
   at Google.Cloud.Diagnostics.Common.EventTarget.ForLogging(LogTarget logTarget, String logName, LoggingServiceV2Client loggingClient, MonitoredResource monitoredResource)
   at Google.Cloud.Diagnostics.Common.ErrorReportingContextExceptionLogger.Create(String projectId, String serviceName, String version, ErrorReportingOptions options)
   at Google.Cloud.Diagnostics.AspNetCore.ErrorReportingExceptionLoggerExtension.AddGoogleExceptionLogging(IServiceCollection services, Action`1 setupAction)
   at XXX.Startup.ConfigureServices(IServiceCollection services) in XXX\Startup.cs:line 35
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()

The relevant C# code is:

services.Configure<StackdriverOptions>(Configuration.GetSection("Stackdriver"));
services.AddGoogleExceptionLogging(options =>
{
    options.ProjectId = Configuration["Stackdriver:ProjectId"];
    options.ServiceName = Configuration["Stackdriver:ServiceName"];
    options.Version = Configuration["Stackdriver:Version"];
});

[jskeet]

@bgrainger: Yes, that sounds like it's the same issue. Can I ask how regularly/reliably you're seeing this? (Any way of provoking it more often would be very welcome.)

I appreciate everyone's patience on this. I expect to make this my main work item next week, seeing as the previous attempts didn't fix it.

[vuukle]

I’ve noticed that re-publishing via power shell fixes the issue but to confirm I will try to only use power shell to see if the issue happens again or not

[bgrainger]

@jonskeet it's happened every single time I've run gcloud app deploy (which is three times in total) since adding the Stackdriver code. (I added logging and error reporting and tracing all at once so I can't specifically narrow it down to one cause, except from the call stack. But I could reduce it to one to simplify the repro.)

[jskeet]

@bgrainger: That's useful to know, thanks. On Monday I'll try following exactly what you've done and see whether that reproduces it for me.

[bgrainger]

I just tried to recreate the problem in a new ASP.NET Core project, but was unsuccessful. I then went back to the project where I was having problems and gcloud app deploy deployed successfully (with Stackdriver logging, error reporting, and tracing). (I didn't change any gcloud settings, so I was redeploying to the same App Engine project.) At this point it's "working" for me, but I'm unsure if I did anything to change it, or if it's an intermittent problem that I'm just no longing hitting.

[jskeet]

@bgrainger: I strongly suspect it's just intermittent. Will be looking into it more tomorrow.

[jskeet]

Okay, some progress of sorts:

• This morning we saw the error multiple times with an earlier version of Google.Apis.Auth
• We checked the corresponding Java code and found that it uses the IP address rather than the name, so we prototyped a change in Google.Apis.Auth to match
• We've still seen one example of that failing - we're trying to reproduce it more to find out timing etc
• We've noticed that the Java code performs 3 attempts, each with a 500ms timeout.

[jskeet]

There's now a new release: Google.Apis.Auth v1.32.2.

This has a per-request timeout of 500ms, but it makes up to 3 requests. It also uses the IP address rather than the name. This now matches the Java implementation.

I'm going to close this issue yet again - but obviously, I'll reopen it if 1.32.2 turns out not to help for whatever reason. All the testing we've performed suggests it should be absolutely fine, but this really is a very temporamental issue to fix...

[helgejfo]

Thanks Jon, appreciate yor effort on this one.

[vuukle]

I pushed my application to production and i am getting this error
Grpc.Core.RpcException: Status(StatusCode=Unauthenticated, Detail="Deadline Exceeded")
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at Grpc.Core.Internal.AsyncCall2.UnaryCall at Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[TRequest,TResponse] at Grpc.Core.Internal.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse] at Google.Cloud.Datastore.V1.Datastore.DatastoreClient.RunQuery at Google.Api.Gax.Grpc.ApiCall.<>c__DisplayClass0_02.b__1
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass1_02.<WithRetry>b__0 at Google.Cloud.Datastore.V1.QueryStreamer.<Sync>d__7.MoveNext at System.Collections.Generic.List1.AddEnumerable
at System.Linq.Enumerable.ToList[TSource]
at Google.Cloud.Datastore.V1.LazyDatastoreQuery.GetAllResults
at VuukleCore.Logic.Impl.BaseCrudService`1.GetByProperties

[jskeet]

@vuukle: That sounds like it's at least a different problem by that point. I'm not quite sure why it would be a status code of Unauthenticated with a detail of Deadline Exceeded... could you raise a different issue for that, with any more details?

(I'm on vacation at the moment, so I won't be terribly responsive until next week, but I'll do what I can.)

[michivo]

I guess I had this problem with an ASP.net Core/App Engine Flex application, too. 9 out of 10 deployments failed, always with the same error message as described above upon startup:
System.InvalidOperationException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. at Google.Api.Gax.TaskExtensions.WaitWithUnwrappedExceptions(Task task) at Google.Api.Gax.Grpc.ChannelPool.GetChannel(ServiceEndpoint endpoint) at Google.Cloud.Diagnostics.AspNetCore.CloudTraceExtension.AddGoogleTrace(IServiceCollection services, Action1 setupAction)
at MyAspNetProject.Startup.ConfigureServices(IServiceCollection services) in C:PathStartup.cs:line 36`

After updating to Google.Apis.Auth v1.32.2, I could successfully deploy 3 times in a row, which is unprecedented.

[chrisdunelm]

@michivo Thanks for letting us know. It's nice to get confirmation that the changes in v1.32.2 have been successful (at least for the issue you were experiencing).