Google-api-dotnet-client: Started to get "A task was cancelled"

[olaj]

I've noticed that i have started to get "A task was cancelled" errors from some tasks im using to collect data from GA. I have read this thread https://github.com/googleapis/google-api-dotnet-client/issues/1394 that looks similar but i don't think i have any network issues. At least it fails both when i run it locally AND in my Azure function. It also doesn't fail all the time, i use a loop to grab alot of data with something like X number of rows in each request. Sometimes it runs for 1 iteration, sometimes 3 and so on.

I suppose it is some kind of timeout error?

• Is it possible to get any more details on the error than "A task was cancelled"? Can i enable more logging somehow?
• What else can i do to debug this?

The gist of my code looks like this.

            while (rowcount >= maxResults && interations < maxIterations)
            {
                var accountEmailAddress = Environment.GetEnvironmentVariable("GA.AccountEmail");
                var api = new GoogleAnalyticsAPI(xmlKey, accountEmailAddress);

                string[] dimensions = new string[] { dimension };
                string[] metrics = new string[] { metric };
                string[] filters = new string[] { $"{pathFilter};{metric}>{viewFilter}" };
                string[] sort = new string[] { "-" + metric };

                log.LogInformation($"Start GA request for iteration {interations} of {maxIterations}.");

                var request = api.BuildAnalyticRequest("ga:15796540", dimensions, metrics, filters, maxResults, sort, startDate, endDate, index, "gaid::-1");

                var response = request.Execute();

                foreach (List<string> row in response.Rows)
                {
                    allRows.Add(row);
                }

                rowcount = response.Rows.Count;

                log.LogInformation($"Collected {rowcount} from GA in iteration {interations}.");

                index += rowcount;
                interations += 1;
            }

        public DataResource.GaResource.GetRequest BuildAnalyticRequest(string profileId, string[] dimensions, string[] metrics,
                                                                    string[] filters, int maxResults, string[] sort,
                                                                    string startDate, string endDate, int startIndex, string segment)
        {

            DataResource.GaResource.GetRequest request = Service.Data.Ga.Get(profileId, startDate,
                                                                                endDate, string.Join(",", metrics));
            request.Dimensions = string.Join(",", dimensions);
            request.MaxResults = maxResults;
            request.Filters = string.Join(",", filters);
            request.Sort = string.Join(",", sort);
            request.StartIndex = startIndex;
            if (segment != "")
            {
                request.Segment = segment;
            }

            return request;

        }

        public GoogleAnalyticsAPI(string xmlKey, string accountEmailAddress)
        {

            var rsa = new RSACryptoServiceProvider();
            RSACryptoServiceProviderExtensions.RSACryptoServiceProviderExtensions.FromXmlString(rsa, xmlKey);


            ServiceAccountCredential credentials = new ServiceAccountCredential(
            new ServiceAccountCredential.Initializer(accountEmailAddress)
            {
                Scopes = new[] { AnalyticsService.Scope.AnalyticsReadonly },
                Key = rsa
            });

            Service = new AnalyticsService(new BaseClientService.Initializer()
            {
                HttpClientInitializer = credentials,
                ApplicationName = "WorthlessVariable"
            });
        }

When i started debugging this i was using _Google.Apis.Analytics.v3 1.37.0.1306_ but i updated it to 1.49.0.1679 but i still get the same error.

[jskeet]

(Quick note: all the maintainers are currently on vacation, and won't be back until January 5th. This is more of a drive-by comment than anything else.)

My guess is that the request is just timing out, as you suspect too. I would recommend that you add some logging for how long it takes, both when it succeeds and when it fails. We can then see whether all the failing ones are failing after the same amount of time.

[olaj]

Ahh.

I've changed my code to this and it seems to work if i let it retry on failed requests. It seems to timeout after 100 sec. Any idea why this has started to timeout all of a sudden?

                {

                    var response = request.Execute();

                    foreach (List<string> row in response.Rows)
                    {
                        allRows.Add(row);
                    }

                    rowcount = response.Rows.Count;

                    log.LogInformation($"Collected {rowcount} from GA in iteration {interations} in {stopwatch.Elapsed.TotalMilliseconds}ms.");
                }
                catch (Exception ex)
                {
                    failCount++;

                    if(failCount < 5)
                    {
                        log.LogError($"GA request FAILED {failCount} time (after {stopwatch.Elapsed.TotalMilliseconds}ms) for iteration {interations} of {maxIterations} with message {ex.Message}, try again.");
                        continue;
                    } else
                    {
                        log.LogError($"GA request FAILED {failCount} time (after {stopwatch.Elapsed.TotalMilliseconds}ms) for iteration {interations} of {maxIterations} with message {ex.Message}, give up.");
                        break;
                    }
                }

[2020-12-29T09:38:05.775Z] GA request FAILED 1 time (after 100117,5431ms) for iteration 0 of 15 with message A task was canceled., try again.
[2020-12-29T09:38:05.778Z] Start GA request for iteration 0 of 15.
[2020-12-29T09:38:06.928Z] Collected 10000 from GA in iteration 0 in 1148,5439ms.
[2020-12-29T09:38:06.929Z] Start GA request for iteration 1 of 15.
[2020-12-29T09:39:46.997Z] GA request FAILED 2 time (after 100066,5789ms) for iteration 1 of 15 with message A task was canceled., try again.
[2020-12-29T09:39:47.001Z] Start GA request for iteration 1 of 15.
[2020-12-29T09:39:48.738Z] Collected 10000 from GA in iteration 1 in 1734,361ms.
[2020-12-29T09:39:48.742Z] Start GA request for iteration 2 of 15.
[2020-12-29T09:39:49.728Z] Collected 10000 from GA in iteration 2 in 981,9855ms.
[2020-12-29T09:39:49.733Z] Start GA request for iteration 3 of 15.

[jskeet]

I don't know of any new timeouts. The way I see it there are three possibilities:

• The timeout itself hasn't changed, but previously you weren't running into it
• The timeout has changed on the server side
• The timeout has changed on the client side

It does seem like the requests that aren't timing out are finishing really quickly (within 2 seconds). Are these the same requests that are timing out?

I can try to find out more information from the Analytics team when I'm back from vacation.

[olaj]

Yeah, they are exactly the same. I'm just running the exact same request again. Haven't made any "deep debugging" though. But it seems totally random. I have also seen the timeouts when i grab very little data.

No stress, seems like the job is working now.

I noticed now that this started to happen when i deployed some new code about a month ago, i didn't touch this code back then but I may have updated some dependencies then but this is a pretty old and "messy" project so it's kind of hard to look into what exactly changed.

It would probably help to see some kind of logs on the other end because of the randomness of the issue.

[jskeet]

Thanks for the updated information. I believe the default HttpClient timeout is 100 seconds, and I suspect that's what's timing out - so my guess is that it's the time taken for the request that's changed. I doubt that that would be affected by the client-side code, unless you've changed what's in the requests.

[Robert-Zandberg]

We have currently the same issue (A task was cancelled).

We're using the Service account method with an certificate.
The code looks like this:

  var certificate2 = new X509Certificate2(@"key.p12", "notasecret", X509KeyStorageFlags.Exportable);
  var credential = new ServiceAccountCredential(new ServiceAccountCredential.Initializer(_gaMail)
  {
    Scopes = scopes
  }.FromCertificate(certificate2));

  var service = new AnalyticsService(new BaseClientService.Initializer
  {
    ApplicationName = "Analytics portal",
    HttpClientInitializer = credential
  });



  var dataList = _db.GaProperties.Where(property => property.Enabled).ToList();
  var dataAray = new GaSiteData[dataList.Count + 1];
  var total = 0;

  try
  {
    var data = service.Data.Realtime.Get("ga:" + dataList[0].PropertyId, "rt:activeUsers").Execute();
    var  gaVisitors = data.TotalsForAllResults["rt:activeUsers"];
    total += int.Parse(gaVisitors);
  }
  catch (GoogleApiException e)
  {
    var gaVisitors = "Error realtime api google";
  }

The error doesn't show up all the time so my guess is that when I build the workaround that retries the failed request it will do the job.

[vso777]

Same issue here...

Our service runs on the first day of the month for about 1.000 requests:

• In November there was no problem with the requests.
• In December about 150 out of 1000 requests ended up with the error message: "A task was cancelled".
• Yesterday I started the service again and there are only a few successful requests. Most of them are getting the same error so it seems that currently it's even getting worse.

I can also confirm that the successful requests (which are the completely the same like the broken ones) return after 1-2 seconds and the errors occur after about 100 seconds.

[jskeet]

Thanks for the extra information, folks. It does sound like it's probably something in the service itself - I'll raise it internally on Tuesday.

[jskeet]

Hi folks, thanks for your patience. I've been in touch with the Google Analytics team, and they're actively working on this problem. I'll update this issue again if I hear more.

[jskeet]

Ah, I hadn't spotted in our internal report that there's a public issue associated with this: http://issuetracker.google.com/176032263

I'll close this issue now, as that's the one to watch. (It's a server-side issue, so the client library can't fix it, I'm afraid.)