Framework: Queue worker continues to attempt job after 'attempts' column goes out of range

[cdbeaton]

• Laravel Version: 5.5.40
• PHP Version: 7.2.6
• Database Driver & Version: MariaDB 10.1.31

Description:

I wrote a job which wasn't working due to a typo in a variable name. Laravel's queue worker kept retrying the job and incrementing the "attempts" column in the queue table. When this hit 255, the queue worker began to throw errors because 256 is an out of range value. The following was written to laravel.log:

[2018-06-16 10:00:02] production.ERROR: SQLSTATE[22003]: Numeric value out of range: 1264 Out of range value for column 'attempts' at row 1 (SQL: update `queue` set `reserved_at` = 1529143202, `attempts` = 256 where `id` = 569) {"exception":"[object] (Illuminate\\Database\\QueryException(code: 22003): SQLSTATE[22003]: Numeric value out of range: 1264 Out of range value for column 'attempts' at row 1 (SQL: update `queue` set `reserved_at` = 1529143202, `attempts` = 256 where `id` = 569) at /home/username/directory/vendor/laravel/framework/src/Illuminate/Database/Connection.php:664, PDOException(code: 22003): SQLSTATE[22003]: Numeric value out of range: 1264 Out of range value for column 'attempts' at row 1 at /home/username/directory/vendor/laravel/framework/src/Illuminate/Database/Connection.php:483)
[stacktrace]
#0 /home/username/directory/vendor/laravel/framework/src/Illuminate/Database/Connection.php(624): Illuminate\\Database\\Connection->runQueryCallback('update `queue` ...', Array, Object(Closure))
#1 /home/username/directory/vendor/laravel/framework/src/Illuminate/Database/Connection.php(490): Illuminate\\Database\\Connection->run('update `queue` ...', Array, Object(Closure))
#2 /home/username/directory/vendor/laravel/framework/src/Illuminate/Database/Connection.php(423): Illuminate\\Database\\Connection->affectingStatement('update `queue` ...', Array)
#3 /home/username/directory/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2173): Illuminate\\Database\\Connection->update('update `queue` ...', Array)
#4 /home/username/directory/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(287): Illuminate\\Database\\Query\\Builder->update(Array)
#5 /home/username/directory/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(268): Illuminate\\Queue\\DatabaseQueue->markJobAsReserved(Object(Illuminate\\Queue\\Jobs\\DatabaseJobRecord))
#6 /home/username/directory/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(198): Illuminate\\Queue\\DatabaseQueue->marshalJob('default', Object(Illuminate\\Queue\\Jobs\\DatabaseJobRecord))
#7 /home/username/directory/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(244): Illuminate\\Queue\\DatabaseQueue->pop('default')
#8 /home/username/directory/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(220): Illuminate\\Queue\\Worker->getNextJob(Object(Illuminate\\Queue\\DatabaseQueue), 'default')
#9 /home/username/directory/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\\Queue\\Worker->runNextJob('database', 'default', Object(Illuminate\\Queue\\WorkerOptions))
#10 /home/username/directory/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(85): Illuminate\\Queue\\Console\\WorkCommand->runWorker('database', 'default')
#11 [internal function]: Illuminate\\Queue\\Console\\WorkCommand->handle()
#12 /home/username/directory/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#13 /home/username/directory/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#14 /home/username/directory/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#15 /home/username/directory/vendor/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#16 /home/username/directory/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\\Container\\Container->call(Array)
#17 /home/username/directory/vendor/symfony/console/Command/Command.php(252): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#18 /home/username/directory/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#19 /home/username/directory/vendor/symfony/console/Application.php(946): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#20 /home/username/directory/vendor/symfony/console/Application.php(248): Symfony\\Component\\Console\\Application->doRunCommand(Object(Illuminate\\Queue\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#21 /home/username/directory/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#22 /home/username/directory/vendor/laravel/framework/src/Illuminate/Console/Application.php(88): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#23 /home/username/directory/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#24 /home/username/directory/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#25 {main}
"} 

This is not a massive bug as the root of my problem was the error in my job code, but I can't imagine it is intended behaviour for the queue worker to attempt to set a column to an out of bound value.

[themsaid]

I think it was designed so that a reasonable number of tries should be set on the worker as it's quite resource consuming to try a failing job forever, so for example if we solve the DB issue and make the default field size much larger the root problem will remain, which is your worker will keep retrying forever until it hits whatever database field limit is set.

So make sure you set a proper number of tries on your worker (--tries=3 for example).

[cdbeaton]

Thanks for the tip, @themsaid - I'll do this in future. Would it make sense for the queue worker to have a default figure of 255 tries as the database limitation already makes this _de facto_ the case?

[mageshkumaar]

@zcbeaton Let's say the worker stops trying after 255 attempts, logs would not be bombarded with the worker failure error and the anomaly might go undetected. Maybe that's why a default value was not set.

It's always recommended to set the retry value.

[linux019]

the job must has a hard limit limit and don't repeat itself indefinitely as designed. DB is intended to store data and not for limiting of jobs with incorrect behaviour

[Harsukh21]

Hi, i resolve this issue in my live laravel application. just add the following line in your job class

public $timeout = 7200; // 2 hours

[linux019]

Hi, i resolve this issue in my live laravel application. just add the following line in your job class

public $timeout = 7200; // 2 hours

Hi, my team dropped laravel from servers entirely and rewrote everything in Golang and Java as laravel is full of bugs and has crucial issues in multi server architecture. And we were very tired to keep forks of laravel as repository owners had ignored all PRs