Google-api-ruby-client: how to retry batch requests?

[pacuna]

I'm making batch requests to the Analytics API. Sometimes I get userRateLimitExceeded and the retries option doesn't seem to be working.

My code:

    @service.batch do |service|
      elements.each do |element|
        service.get_ga_data(profile,
                            start_date, end_date,
                            metrics,
                            filters: filters(element), 
                            options: {retries: 5},
                           &callback)
      end
    end

And the callback

     callback = lambda do |res, err|
      if err.present?
        puts "error code: #{err.status_code}"
      else
        # do something with the result
      end
    end

I've tried with both, passing the options hash with the retries key and also setting Google::Apis::RequestOptions.default.retries = 5 with no success. The requests with error just get lost.

What am I missing?

Thanks in advance

[sqrrrl]

It _should_ work with:

service.batch(options: { retries: 5 }) do |service|
 ...
end

Setting default option should also work, but it has to be done before the service is created.

Retries on individual operations within a batch isn't yet supported. It's a bit complicated to do that correctly (e.g. extract the failed requests/responses, build a new batch, retry, repeat... merge all the results...)

I'd caution against using retries with batches unless you know the operations are safe to repeat. Since the entire batch is repeated, you may be replaying successful operations as part of it.

[pacuna]

Hey @sqrrrl thanks for answer. So are you saying that if an entire batch fails, only then you can retry? For example in my case I have a batch with 100 requests. If the first ten succeed, then there's no way of retrying the next ones that fail because of quota issues?

Thanks.

[sqrrrl]

Yeah, it's just that I haven't implemented that logic in the client. It's doable, just a lot of work to add custom retry logic for batches to selectively retry the individual parts.

I thought about doing it before and have some ideas how it might be done, it just hasn't been much of a priority. I can re-open as a feature request if you think it would be helpful.

[pacuna]

@sqrrrl Oh ok! Don't worry. It's actually not hard to implement on my side. Just wanted to be sure that I wasn't missing something that may be in the library for this use case.
Thanks a lot.

[sqrrrl]

FYI, I took a quick stab at it yesterday to see how much work it would be and it's not all that bad. A few weird edge cases to account for, but should be able to add it in 0.10