Google-cloud-go: cloudtasks apiv2beta3 "The deadline cannot be more than 30s in the future"

[dougdonohoe]

Client

cloudtasks/apiv2beta3

Describe Your Environment

GAE running go112, migrating from old go1.9 environment, so moving old GAE code to cloud tasks.

Expected Behavior

I can post a task to cloud tasks with this code:

    _, err := client.CreateTask(ctx, &tasks.CreateTaskRequest{
        Parent: t.QueueName,
        Task: &tasks.Task{
            PayloadType: &tasks.Task_AppEngineHttpRequest{
                AppEngineHttpRequest: &tasks.AppEngineHttpRequest{
                    HttpMethod: tasks.HttpMethod_POST,
                    AppEngineRouting: &tasks.AppEngineRouting{
                        Service: "admin",
                    },
                    RelativeUri: "/xxx/status",
                    Headers:     map[string]string{"Content-Type": "application/x-www-form-urlencoded"},
                    Body:        body,
                },
            },
        },
    })

Actual Behavior

All posts fail with a message like this;

rpc error: code = InvalidArgument desc = The request deadline is 2019-09-13T06:45:15.1074040855-07:00. The deadline cannot be more than 30s in the future.

In this above case, the request time was at 2019-09-13T13:44:23.263625162Z. The deadline is 52 seconds in the future. I have no idea where this deadline is getting set.

This seems related to

• https://github.com/googleapis/google-cloud-java/issues/5555
• https://github.com/googleapis/gax-java/pull/745

Notes

I actually want to replicate the delay functionality from the old app engine API:

task := taskqueue.NewPOSTTask("/xxx/status", v)
    task.Delay = 30 * time.Second
    _, err := taskqueue.Add(ctx, task, s.QueueName)

I initially tried using this field in Task:

ScheduleTime: &timestamp.Timestamp{Seconds: delay}

But it doesn't seem to work (or impact the deadline issue described above).

[dougdonohoe]

As background, the system we have posts a task (with delay), then when that task is performed, it may enqueue another task if certain criteria isn't met. It's effectively a polling-type mechanism.

This may be a problem in my code, but I'm not sure yet. I'll add more details as I get more clarity.

[dougdonohoe]

I have a workaround - if I pass in context.Background() to the CreateTask call, I don't run into issues. Previously, I was passing in the ctx of the originating request - not why that causes problems (don't know the internals well enough yet to speculate).

Also, with it working, I have verified that using ScheduleTime also works.

[jadekler]

@dougdonohoe I don't believe that cloudtasks imposes any deadline, but I could be wrong. Just to double check, are you certain the ctx you're passing in doesn't have a deadline on it from somewhere else higher in your stack?

[dougdonohoe]

@dougdonohoe I don't believe that cloudtasks imposes any deadline, but I could be wrong. Just to double check, are you certain the ctx you're passing in doesn't have a deadline on it from somewhere else higher in your stack?

Good question. We do have this sprinkled through our code base, in an attempt to (I assume) ensure any particular endpoint is limited to 60 seconds.

ctx, _ = context.WithTimeout(ctx, 60*time.Second)

So is this not a valid use case? I have code that looks up things in datastore, sometimes calls out to another REST API, and depending on the result, posts a task to the queue. How else can I ensure that doesn't take longer than a minute?

It's not a show stopper, it's just counter-intuitive I guess.

[jadekler]

Gotcha. Yeah, that seems to be the culprit. Most APIs dont really care what deadline you set, but it appears as though this particular API does. So, the context.Background solution seems appropriate, or use cancellation instead of deadlines if you need to cancel requests for some reason.

[finci-square]

referencing this issue i've just posted https://github.com/googleapis/google-cloud-java/issues/6963

I don't specifically set timeout in my code. the suggested workaround to specify GAX 1.47.0 causes my usage of KMS to fail when I call KeyManagementClient.create()

wanted to posts in case there are any thoughts from contributors on this thread

[derekperkins]

This is 100% a bug in the library. In no way shape or form should the context deadline of the caller be silently translated behind the scenes into an invalid argument for cloudtasks. This should be reopened and fixed.

[tbpg]

@derekperkins, that's not so clear to me. One of the purposes of context.Context is to be able to pass deadlines and timeouts around to function and API calls. In this case, when your Context deadline is reached, there is no need to try to continue the API call -- it _should_ be cancelled by the frontend and backend.

Two ways to communicate that to the backend are cancelling the request or including the timeout within the request.

The 30 second limit seems to be set on the server side. So, I don't think there is a good way for us to modify the client to do min(30 seconds, currentTimeout).

[derekperkins]

If the context times out, there are already 2 error codes for that - Canceled or DeadlineExceeded. What makes this error so pernicious is that an error is being caused for a field that doesn't exist in the proto and that isn't documented anywhere.

We've been heavy GCP users for 6 years now and there isn't a single other api that rejects a request because the deadline is too far in the future, that just doesn't make any sense. I get now that it's a server problem, and so I still believe that an issue should be raised internally to fix this since there is no reason for it to exist.

I propose that this get solved in 1 of 2 ways in this client package:

1. Manually set the context timeout to be compliant, context.WithTimeout(ctx, 30 * time.Second). The context package will handle the min() behavior you were thinking about. This should solve the problem for all users with no side-effects other than deriving one extra context.
2. If for whatever reason you aren't allowed to deviate from server behavior that way, detect the timeout yourselves in the package and return a meaningful error.

[tbpg]

cc @codyoss @noahdietz (about to be unavailable).

[benjaminjkraft]

Will this be reopened? I agree with @derekperkins that it should be a bug. We ran into it (when we set a deadline on context way higher up in the stack) and it was very much a surprise that a too-loose deadline would cause an RPC to error. Normally you expect that a huge deadline will just be effectively ignored (or truncated to a lower value). We're going to implement (1) in our code, but it seems like the library should just do it.