Google-cloud-go: bigquery: Need support to insert jobs for asynchronous queries.

[tswast]

I couldn't figure out how to insert a job in order to run an asynchronous query (for longer-running queries). See the Python sample for this: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/bigquery/api/async_query.py

Job has Read but no Insert. https://godoc.org/cloud.google.com/go/bigquery#Job

[tswast]

I believe this is needed for long-running queries in order to avoid connection timeouts.

[broady]

I think, generally, we don't have good support for long running operations in these clients.

It probably deserves quite a bit of thought across different packages.

[derekperkins]

The point is to not have to have a long running client. The job insert puts it in asynchronously and gives you back a job ID that you have to poll for completion.

[broady]

@derekperkins we call them "long-running operations" (they're represented by an Operation type in the protos).

Though I think BigQuery predates those, and calls them jobs.

[jba]

Use Client.Copy to insert a Job.

[tswast]

No, that's not quite what I want. https://godoc.org/cloud.google.com/go/bigquery#Client.Copy A copy job is a kind of job, but it won't run a long-running query for me.

[derekperkins]

This is what he's looking for. We use it for most of our queries too.
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.query

[derekperkins]

@broady Thanks for the clarification. That's a weird thing to define, since the api call for that is short, but represents a long running operation.

[mcgreevy]

@tswast The library actually supports what you want, but it sorely needs better documentation. The Client.Copy method is used for all long-running operations. It is named to be analogous to the builtin Go copy function, rather than the bigquery table-to-table copy operation. We may need to reconsider this name as it's easily confused with the bigquery copy operation.

The Client.Copy method takes a Source and a Destination as parameters, and returns a Job. If you pass a Query as a Source and a Table as a Destination, then you will initiate a long-running query. If you pass a Table as a Source and a Table as a Destination, you will initiate a copy. If you pass a GCSReference as a Source and a Table as a Destination, you will initiate a load. etc.

To perform a long-running query, you should do something like this:

    client, err := bigquery.NewClient(ctx, "project")
    if err != nil {
        log.Fatalf("Creating bigquery client: %v", err)
    }

    outputTable := client.Dataset("dataset").Table("table")
    query := client.Query("select * from blah")

    // Query data.
    job, err := client.Copy(ctx, outputTable, query, bigquery.WriteTruncate)

You can then poll the returned job (Job.Status) until it's done, or just call Job.Read which returns an Iterator, which will handle the polling for you.

[jba]

@mcgreevy Can you make that snippet into an Example?

[jba]

Closing this, since we do support the desired functionality.

[tswast]

Thanks @mcgreevy I will try that. The documentation seems very incorrect if this is the case. https://godoc.org/cloud.google.com/go/bigquery#Client.Copy very much makes it sound like a table copy operation, only.

[redwrasse]

As mentioned above, can someone please add to the documentation examples the above use of Client.Copy for performing a long-running query? (This was already asked in an above comment but didn't make it into the documentation). I've spent a couple hours hunting around on the internet trying to figure out how to do this with the Go library.

[jba]

The client surface has changed considerably since then. It should be much clearer. In fact, the package godoc begins with a section on querying that covers synchronous and asynchronous queries, and even how to retrieve query results from a different process.

You may want to go get -u cloud.google.com/go/bigquery and try it again. If you still have questions, feel free to open a new issue. (Issues are fine for questions, as well as bugs and feature requests.)