Framework: artisan queue:restart doesn't actually restart worker daemons, just kills them

[mattzuba]

If I open two terminal windows and run ./artisan queue:work --daemon in one, and then run ./artisan queue:restart in the other, the first one just exits and that's it. Digging through the code for queue workers, daemonizing and restarting, there is nothing that actually restarts a queue worker. The Restart command updates a cache value, the worker reads it, then goes to the stop() method where a 'die' statement occurs.

[GrahamCampbell]

What laravel version?

[mattzuba]

I'm using 4.2 right now, but I looked at the code for 5 and it appears to
work in the same fashion.
On Jan 8, 2015 2:51 PM, "Graham Campbell" [email protected] wrote:

What laravel version?

—
Reply to this email directly or view it on GitHub
https://github.com/laravel/framework/issues/6950#issuecomment-69255974.

[Garbee]

Why are you trying to have two daemons watch the same queue?

[GrahamCampbell]

@Garbee Why not?

[Garbee]

I thought that only one thing could watch a given queue/tube. At least, when I tested multiple listeners with a beanstalk tube it would only allow one to work at a time. So whether I have one listener or 4 only one queue item is being worked on at a time. (Although I could just have things configured incorrectly.)

[mattzuba]

While I'm not using multiple daemons/workers here (yet, anyway), it's quite
common to use multiple workers on a queue.

Imagine a queue that holds videos for processing. You wouldn't want only
one queue worker processing those videos, you'd want as many as your system
could handle to process them as fast as possible.
On Jan 10, 2015 7:24 AM, "Jonathan Garbee" [email protected] wrote:

I thought that only one thing could watch a given queue/tube. At least,
when I tested multiple listeners with a beanstalk tube it would only allow
one to work at a time. So whether I have one listener or 4 only one queue
item is being worked on at a time. (Although I could just have things
configured incorrectly.)

—
Reply to this email directly or view it on GitHub
https://github.com/laravel/framework/issues/6950#issuecomment-69457468.

[GrahamCampbell]

@mattzuba For now, ironmq might be a solution for you.

[mattzuba]

I managed to get it working yesterday by putting a respawn stanza in my
upstart file and that keeps it running when it's killed. That'll work for
me for now.

I think what laravel needs is to create a queue:daemon command that acts as
a process manager to spawn workers. Though in my case for the way elastic
beanstalk works with deployments, that still wouldn't work right, but itd
be a great addition and step in the right direction.
On Jan 10, 2015 9:03 AM, "Graham Campbell" [email protected] wrote:

@mattzuba https://github.com/mattzuba For now, ironmq might be a
solution for you.

—
Reply to this email directly or view it on GitHub
https://github.com/laravel/framework/issues/6950#issuecomment-69460646.

[Garbee]

So, you aren't using multiple daemons yet. Isn't that exactly the problem
you are describing in the initial issue?
On Jan 10, 2015 10:28 AM, "mattzuba" [email protected] wrote:

While I'm not using multiple daemons/workers here (yet, anyway), it's
quite
common to use multiple workers on a queue.

Imagine a queue that holds videos for processing. You wouldn't want only
one queue worker processing those videos, you'd want as many as your
system
could handle to process them as fast as possible.
On Jan 10, 2015 7:24 AM, "Jonathan Garbee" [email protected]
wrote:

I thought that only one thing could watch a given queue/tube. At least,
when I tested multiple listeners with a beanstalk tube it would only
allow
one to work at a time. So whether I have one listener or 4 only one
queue
item is being worked on at a time. (Although I could just have things
configured incorrectly.)

—
Reply to this email directly or view it on GitHub
https://github.com/laravel/framework/issues/6950#issuecomment-69457468.

—
Reply to this email directly or view it on GitHub
https://github.com/laravel/framework/issues/6950#issuecomment-69459465.

[mattzuba]

No, the problem I'm describing is that the queue:restart command doesn't
actually restart the deamonized workers, it causes the workers to simply
die after their current job. A process manager like upstart or supervisord
is required to make sure they stay running.
On Jan 10, 2015 11:07 AM, "Jonathan Garbee" [email protected]
wrote:

So, you aren't using multiple daemons yet. Isn't that exactly the problem
you are describing in the initial issue?
On Jan 10, 2015 10:28 AM, "mattzuba" [email protected] wrote:

While I'm not using multiple daemons/workers here (yet, anyway), it's
quite
common to use multiple workers on a queue.

Imagine a queue that holds videos for processing. You wouldn't want only
one queue worker processing those videos, you'd want as many as your
system
could handle to process them as fast as possible.
On Jan 10, 2015 7:24 AM, "Jonathan Garbee" [email protected]
wrote:

I thought that only one thing could watch a given queue/tube. At
least,
when I tested multiple listeners with a beanstalk tube it would only
allow
one to work at a time. So whether I have one listener or 4 only one
queue
item is being worked on at a time. (Although I could just have things
configured incorrectly.)

—
Reply to this email directly or view it on GitHub
<
https://github.com/laravel/framework/issues/6950#issuecomment-69457468>.

—
Reply to this email directly or view it on GitHub
https://github.com/laravel/framework/issues/6950#issuecomment-69459465.

—
Reply to this email directly or view it on GitHub
https://github.com/laravel/framework/issues/6950#issuecomment-69465419.

[valorin]

My understanding is that there is no easy, or nice, way to get PHP to do this. So the queue:restart just kills it, and relies on some external system to bring it back up. Given that PHP isn't designed for running and monitoring daemons, it sounds like a sensible way to do it. The confusion is in the name queue:restart, which implies more than it actually does without setup from other tools.

I personally just use run-one on Ubuntu in a cron to ensure the daemon is running. It avoids the need to configure and manage a tool like upstart with root privileges, when the www-data user can edit their own cron easily.

I.e:

* * * * * run-one php /var/www/project/artisan queue:work --daemon 1>> /dev/null 2>&1

[GrahamCampbell]

A common thing people do us to use supervisor to monitor the queue workers.

[ux-engineer]

@valorin interesting spin that a simple cron line could do the same trick than Supervisor – what's the catch, why is not cron mentioned as an alternative for process manager tools in the doc?

UPDATE: Oh, I see now...run-one is a separate utility. But that would still be simpler setup than with Supervisor.

[valorin]

@envision I think it's the case that using a tool like run-one with cron is a lot less well known than using a tool like Supervisor. Supervisor is also designed for that task, while run-one is more of a hack... even if it is a lot simpler. :-)

[powelski]

I would be happy to see the queue:restart renamed to queue:kill or similar.

[zmonteca]

@valorin I'm not sure I'm totally clear on the idea of the run-one in the crontab.

Will this start and run the artisan cmd, as well as make sure if the process stops or is killed that it keeps running or restarts the artisan queue:work?

I'm running:

* * * * * run-one php /var/www/artisan queue:work --daemon 1>> /dev/null 2>&1

And cron doesn't seem to ever start the process no matter what I do.

[valorin]

@zmonteca it checks if the script is running, and if not, starts it up. It won't start multiple and won't stop existing to start new ones. I run it every minute so if the command finishes/crashes, it'll start it up again the next time the minute ticks over.

It's possible your script has hung and just isn't processing anything. Maybe try piping the output to a file and seeing what it's saying. Also try something like ps aux | grep artisan to check if it's working or not.

[zmonteca]

@valorin yeah, something is off because if I kill the queue manually the run-one cron job doesn't seem to start it back up again. I can't seem to figure out why.

I LOVE THIS SOLUTION, FWIW. I just need to figure out how to make it work reliably :)

[zmonteca]

@valorin it's an error with cron:

Mar 25 15:10:03 localhost cron[30822]: Error: bad username; while reading /etc/crontab
Mar 25 15:10:03 localhost cron[30822]: (*system*) ERROR (Syntax error, this crontab file will be ignored)

Did you encounter this using the run-one approach?

I was able to get the cron restarted without errors. The syntax working for me on Ubuntu 14.04 is:

* * * * * root run-one php /var/www/vhosts/dunbararmoredinvoice.com/artisan queue:work --daemon 1>> /dev/null 2>&1

[valorin]

@zmonteca Oh! Yeah, you're missing the user declaration in your cron command above. The root is missing from before. I'd recommend running it as your web user though, most likely www-data. As root gives full access to the server and may break file permissions (or worse).