Google-cloud-go: BigQuery over gRPC

[derekperkins]

Are there plans to do BigQuery streaming over gRPC? Right now it appears to be creating an http client - https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/bigquery/bigquery.go#L54.

For the use case, I'm thinking about streaming latency. Right now I'm using my own http client and it takes ~250ms minimum to insert rows and I'm trying to decide if/when to switch to using this package. What kind of latency should I be expecting?

[jba]

@jgeewax re BigQuery plans.

Not sure what the expected latency should be, and it might depend on your schema. But it's unlikely this package would be slower (or faster) than your homegrown one. If it is slower, let us know what you're doing and we'll try to incorporate it.

[derekperkins]

Is it doing anything to persist the connection beyond standard http keep alives? Is it using JSON or AVRO to send the data across the wire?

[jba]

JSON is the serialization format. I believe it uses the standard http.DefaultClient. @bradfitz can probably tell you more.

[bradfitz]

It'll be speaking be speaking http2.

If you don't know where your 250ms is doing, you need to understand that first before you change anything.

[derekperkins]

I was just hoping that since all the other libraries were switching to be gRPC backed that BigQuery was going too as well. I've seen how well it performs with Bigtable, and it seems like there's a lot to be gained from not using json as the serialization format. Thanks for the feedback.

[nightlyone]

We currently have a workload of real-time streaming of rows into BigQuery which is dominated by the JSON marshaling. GRPC would be a huge speed-up and cost reduction for us here.

[derekperkins]

Does BigQuery just not have a gRPC endpoint like the other services, so it isn't even an option for this package?

Edit: Added gRPC issue to BQ tracker https://code.google.com/p/google-bigquery/issues/detail?id=773

[jba]

Apparently it does not.

[pongad]

BigQuery doesn't support gRPC endpoint, so I think this isn't an issue for the client library repo. If this is still needed, let's communicate on the issue @derekperkins linked? I'll close this for now, but please reopen if you think it belongs here.

[AFMiziara]

Is there any plans nowadays to incorporate gRPC to BigQuery?