Horizon: Timeout for long running task

[mvieghofer]

Hi,
I have a task that will take a few minutes to complete. I've already set the retry_after value in the config/queue.php to 3600 but my task is still canceled after 60 seconds.

I don't get an error message or something like this. The task just stops working after 60 seconds and keeps marked as running in the dashboard.

Is there a way to increase the timeout for a task?

Maybe this issue is related to #254

[mvieghofer]

I already found a solution to my problem.

In the config/horizon.php file I created a second supervisor for long running tasks and set the timeout for this supervisor to 900 seconds (15 minutes). Then I created a new queue config with a retry_after value of 1200 seconds (20 minutes). I also configured the supervisor to use this redis queue config.

To be more specific my config files now look like this:

config/horizon.php

...
'environments' => [
        'production' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => [
                    'default'
                ],
                'balance' => 'simple',
                'processes' => 9,
                'tries' => 2,
            ],
            'supervisor-long-running' => [
                'connection' => 'redis-long-running',
                'queue' => [
                    'long-running-queue'
                ],
                'balance' => 'simple',
                'processes' => 9,
                'tries' => 2,
                'timeout' => 900,
            ],
        ],
]
...

config/queue.php

...
'connections' => [
        ...
        'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => 'default',
            'retry_after' => 60,
        ],
        'redis-long-running' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => 'default',
            'retry_after' => 1200,
        ],
    ],
...

With this configuration my long running tasks works. All tasks are will be done within a few (milli)seconds will be processed using the default queue and therefore the default timeout.

[codercms]

@mvieghofer thanks for the solution

[Donny5300]

Thanks for the solution!

[aarondfrancis]

Thanks!

[Stereotip42]

Thanks

[bilfeldt]

@mvieghofer This was a nice solution. Might I ask how you are scheduling your long running tasks to run on you new redis-long-running connection?

[mvieghofer]

@bilfeldt You can do this using the onQueue('long-running-queue') method you can call on the result you get from the dispatch method.

A full example would be MyDispatchableClass::dispatch($event)->onQueue('long-running-queue').

[bilfeldt]

@mvieghofer My understanding is that your proposal will run the job on the queue='long-running-queue' but not on the connection redis-long-running or am I mistaken?

Normally you would like your regular "fast" tasks and jobs to run on the default queue which you defined in your redis-connection. Now we have some really long running ("slow") jobs which timeouts, for example, a job processing a Podcast. With your solution then we create a new redis-long-running-connection with a longer timeout and a new Horizon superviser for the default queue on that new connection.

For a job to run on the default queue of the redis-long-running-connection, then we use:

ProcessPodcast::dispatch($podcast)->onConnection('redis-long-running');

See: Dispatching To A Particular Connection

But how is it possible when scheduling a job in the app/Console/Kernel.php to run on a specific queue of a given connection? I am wondering if it's better to set a specific timeout for the long-running jobs directly on the job? See https://laravel.com/docs/5.6/queues#max-job-attempts-and-timeout

Thank you for the feedback :)

[mvieghofer]

@bilfeldt Actually we do as I described above. We dispatch an event on a dispatchable class on a specific queue.

If we have the following class:

namespace Test;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;

class DispatchableClass implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable;

    private $data;

    public function __construct(array $data)
    {
        $this->data = $data;
    }

    public function handle()
    {
        // Do something here
    }
}

We can dispatch an Event for this job like this:

$data = [];
DispatchableClass::dispatch($data)->onQueue('long-running-queue');

If the handle method finishes within the defined timeout you shouldn't have any problems.

[jandante]

@mvieghofer Thanks for this solution.

Would be nice though that you can set a timeout property when you launch the job.

[josegus]

so, basically adding the "timeout" key in horizon.php should be enough

[squatto]

so, basically adding the "timeout" key in horizon.php should be enough

While technically enough, yes, you run the risk of masking problems with your jobs that should run quickly. That's why it's good to split them out into your regular queue and your long running queue: long running jobs are given plenty of time to run, and regular jobs still fail if they go beyond the default timeout.

[squatto]

Replying to @bilfeldt:

My understanding is that your proposal will run the job on the queue='long-running-queue' but not on the connection redis-long-running or am I mistaken?

If I'm understanding this correctly, as he has it configured, his redis connection handles default queue jobs, and his redis-long-running connection handles long-running-queue queue jobs. That's why you don't have to specify the _connection_ when you queue the job - you're specifying a queue that's only being handled by that specific connection. You would need to specify the connection if you had queues with the same names on both connections. I think ;) I'm still not 100% sure I grok the connection-vs-queue concept either, especially when you throw Horizon into the mix.

[bilfeldt]

@squatto I think your assumption is valid. Check the Laravel Documentation section Connections Vs. Queues:

Before getting started with Laravel queues, it is important to understand the distinction between "connections" and "queues". In your config/queue.php configuration file, there is a connections configuration option. This option defines a particular connection to a backend service such as Amazon SQS, Beanstalk, or Redis. However, any given queue connection may have multiple "queues" which may be thought of as different stacks or piles of queued jobs.

Also, note that Horizon has a _Queue Wait Time Thresholds_ setting called waits in config/horizon.php where it says:

Every connection / queue combination may have its own, unique threshold (in seconds) before this event is fired.

    /*
    |--------------------------------------------------------------------------
    | Queue Wait Time Thresholds
    |--------------------------------------------------------------------------
    |
    | This option allows you to configure when the LongWaitDetected event
    | will be fired. Every connection / queue combination may have its
    | own, unique threshold (in seconds) before this event is fired.
    |
    */
    'waits' => [
        'redis:default' => 60,
    ],

As I understand it:

• Laravel can have multiple connections (like Amazon SQS, Beanstalk, or Redis)
• Each connection has at least one queue, but can also have more
• Horizon can have multiple _queue workers_
• Each queue worker uses one (and only one) connection
• Each queue worker works on at least one (but can be more) of the queues that are associated with the connection
• retry_after is a parameter for the connection
• timeout is a parameter for the queue
• retry_after must always be larger than the timeout

[mityukov]

If I define $timeout on the job Class itself — will it override connection/queue settings?

[safjammed]

@bilfeldt Actually we do as I described above. We dispatch an event on a dispatchable class on a specific queue.

If we have the following class:

namespace Test;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;

class DispatchableClass implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable;

    private $data;

    public function __construct(array $data)
    {
        $this->data = $data;
    }

    public function handle()
    {
        // Do something here
    }
}

We can dispatch an Event for this job like this:

$data = [];
DispatchableClass::dispatch($data)->onQueue('long-running-queue');

If the handle method finishes within the defined timeout you shouldn't have any problems.

That doesn;t work. My queue wont process. It only works when i start queue with php artisan queue:work --queue=redis-long-runnig

[bilfeldt]

That doesn;t work. My queue wont process. It only works when i start queue with php artisan queue:work --queue=redis-long-runnig

@safjammed I assume you mean that the job is not being processed by Horizon. Have you made sure to make the necessary queue configuration and horizon configuration as stated at the top: https://github.com/laravel/horizon/issues/255#issuecomment-349621321