Google-cloud-ruby: BigQuery: Retry on 'internalError' codes

[ags]

Is your feature request related to a problem? Please describe.

The BigQuery documentation suggests to troubleshoot an error with the internalError code you should 'retry the operation', but the service backoff mechanism only automatically retries on rateLimitExceeded and the similar, but apparently not the same, backendError codes.

https://cloud.google.com/bigquery/docs/error-messages#errortable

https://github.com/googleapis/google-cloud-ruby/blob/f9b6bf8fbc9f887712e0572322077fb24d61d71b/google-cloud-bigquery/lib/google/cloud/bigquery/service.rb#L539

Describe the solution you'd like

To include internalError in the set of retryable errors.

Describe alternatives you've considered

The set of reasons can't be easily changed by consumers of the gem, but that's another option, though if the gem retries backendError and rateLimitExceeded without additional config it seems like it should handle internalError too.

Consumers could also wrap their API calls in another retry mechanism, but that doesn't seem like a good user experience given the gem already has retry behaviour built in.

Additional context

None

[shollyman]

Thanks for reporting this.

There is conflicting information published about retry policies, and some of the guidance is very context-dependent. I'll see about revisiting that guidance in the error handling documentation to clarify this better (filed this as issue 169673586 internally).

While we've considered having custom retry predicates in the library, we've not moved forward with it as there are certain cases that necessitate user-level retries, as the library cannot effectively determine whether certain failed operations are safe to retry (ex: DML queries that mutate queries, but fail due to concurrent modification).

It is still the case that internalError is not appropriate for inclusion in the general retry predicate, but I'll see about addressing this in the course of resolving 169673586.

[quartzmo]

Closing this issue as currently not actionable in this client library. We can reopen if appropriate when there is progress on the internal issue. Thanks @ags!