Cms: Proposal: Async (Background) Queue

[ostark]

First off, using the new yii2-queue is a great improvement!

But the HTTP/ajax based queue is still the default, right? What's wrong with it? In PHP FPM environments, when all PHP FPM processes are blocked by ajax requests, the site can become unresponsive. Fortunately, we can turn off runQueueAutomatically and run the processes in the background.

However, when we trigger craft queue/run by a cron job, the queue handling is delayed a bit. And for craft queue/listen we may use a process monitor to ensure that the queue listener does not stop running.

Proposal to fix this

If we could run a daemonized php process in the background to handle the jobs right after the job was created - without blocking the rendering of the site. For laravel there is a package that does exactly this.

Since there is no package for Yii so far, I wrote a plugin for Craft 3 - just as a proof of concept:
https://github.com/ostark/craft-async-queue

Please let me know what you think. Is this worth a PR here?

---

This is how the process execution looks like:

Running processes

Empty queue (only php-fpm master is running)

xxx:~$ ps auxf | grep php
root      2953  0.0  0.0 399552 13520 ?        Ss   12:27   0:00 php-fpm: master process (/etc/php/fpm.conf)
````

**New jobs pushed** (php-fpm master + child + bin/php daemon started)

xxx:~$ ps auxf | grep php
root 2953 0.0 0.0 399552 13520 ? Ss 12:27 0:00 php-fpm: master process (/etc/php/fpm.conf)
app 3031 2.2 0.2 718520 45992 ? S 12:31 0:00 _ php-fpm: pool www
app 3033 1.2 0.2 280936 32808 ? S 12:31 0:00 /usr/bin/php craft queue/run -v
app 3034 0.0 0.0 4460 784 ? S 12:31 0:00 _ sh -c /opt/php/7.1/bin/php craft queue/exec "297" "0" "1" --verbose=1 --color=
app 3035 1.2 0.2 280928 32280 ? S 12:31 0:00 _ /opt/php/7.1/bin/php craft queue/exec 297 0 1 --verbose=1 --color=

**No requests** (only php-fpm master + bin/php daemon still running until queue is empty)

xxx:~$ ps auxf | grep php
root 2953 0.0 0.0 399552 13520 ? Ss 12:27 0:00 php-fpm: master process (/etc/php/fpm.conf)
app 3033 0.3 0.2 280936 32844 ? S 12:31 0:00 /usr/bin/php craft queue/run -v
app 3044 0.0 0.0 4460 688 ? S 12:31 0:00 _ sh -c /opt/php/7.1/bin/php craft queue/exec "301" "0" "1" --verbose=1 --color=
app 3045 1.1 0.2 280928 32396 ? S 12:31 0:00 _ /opt/php/7.1/bin/php craft queue/exec 301 0 1 --verbose=1 --color=

```

[angrybrad]

Sorry if I'm being dense, @ostark, but can you achieve the same thing by daemonizing craft queue/listen and passing in 0 for the delay parameter?

[ostark]

Sorry I closed this by accident.

queue/listen would work the same way. However listening to potential jobs does not make much sense, as the jobs get processed when they arrive.

And why do you prefer a delay of 0? This means a sleep(0) in the while loop and can be very expensive for the DB and on the 'php worker' as well.

[ostark]

@takobell do you see the benefits of this approach?

[angrybrad]

@ostark Ahh... I misunderstood. The difference is that queue/listen is constantly polling at a set interval and could be wasted CPU cycles in the case of low queue activity. Your change doesn't poll, but will trigger a queue/run asynchronously/immediately when a new job is added to the queue. Correct?

[ostark]

Right

[angrybrad]

I like it... thoughts @brandonkelly?

[brandonkelly]

Like the concept too. I don’t think we should assume that PHP will be capable of spawning a child process though, so we’d need to do one of these things:

1. Add a new runQueueAsNewProcess config setting. Only checked if runQueueAutomatically is set to true (is by default), and determines whether auto-run queues should happen via Ajax (stays the default behavior) or as a separate PHP process, similar to @ostark’s proof of concept.
2. Replace runQueueAutomatically with a new config setting – maybe called queueMode, with possible values:
   
   
   
   • "ajax": same as current behavior with runQueueAutomatically enabled (default value)
   
   
   • "process": same as proof of concept behavior
   
   
   • "manual" / null / false: same as current behavior with runQueueAutomatically disabled
   
   

[ostark]

Glad you like it @takobell + @brandonkelly 👍

I prefer the second config option queueMode (queueHandler makes even a bit more sense). It may requires a bit more refactoring, but is easier to understand more flexible.

In my example I used craft queue/run. This works, but my intention was to handle a single job. I've tried to run craft queue/exec but I was unsure about the 2nd parameter. Maybe using run is fine, this way all (delayed) jobs get executed at some point.

Instead of making the path to the php binary configurable, we could make use of ProcessPhpExecutableFinder. The PHP_BINDIRconstant is set properly in a common PHP-FPM environment, at least at fortrabbit.com.

[angrybrad]

@ostark Looks like that Symfony component requires PHP 7.1, which we wouldn't be able to require until Yii 2.1. https://github.com/symfony/process/blob/master/composer.json

[ostark]

Nah, only the master branch requires 7.1

https://github.com/symfony/process/blob/3.4/composer.json

You depend on it already (indirectly)
https://github.com/composer/composer/blob/master/composer.json#L33

[khalwat]

This would be fantastic

[joshangell]

I'm liking this idea a lot ...

[phoob]

Wouldn't this solve the issue we were talking about in Slack, @brandonkelly ? I'm all for it 💯

[ostark]

Are you interested in a PR? @brandonkelly @takobell

[brandonkelly]

@ostark The more I think about it, the more I think this would be better off as a plugin. The main reason is because it’s a little awkward that the newly-spawned worker would really just be a normal worker that would also be dealing with other jobs in the queue, not just the one that was just added. And it would be really easy to do as a plugin – it just needs to listen to the queue’s afterPush event, and then kick off a worker process from there. The current runQueueAutomatically config setting would need to be set to false just like you would if the worker was a supervisor’d daemon for the queue/listen command.

[ostark]

little awkward that the newly-spawned worker would really just be a normal worker that would also be dealing with other jobs in the queue, not just the one that was just added.

if this is really an issue, it can be solved by directly calling craft queue/exec instead of craft queue/run.

And it would be really easy to do as a plugin

@brandonkelly Sure, if you are satisfied with the current "frontend queue handler" (I'm not) and "true background queue processing" as an very optional thing, a plugin is better than implementing it in core.

I really liked the second configuration approach. Anyways, I'll put it an a usable plugin.

[narration-sd]

@ostark I've found it usually very fruitful to remember how consistenly mindful the Craft persons have been in reckoning their entire span of designer usage scenarios.

Here, it occurs to me immediately that where Craft is used on any kind of a shared hosting, you won't be permitted to run a worker for more than maximum a few tens of seconds. At such a point, it will likely trip non-advertised watchdogs, whose consequence can be as much as summarily killing in mid-execution every single process and thus web response associated with the site owner/user identity.

I had to track something like this down in the early days of Craft Tasks, and it was not fun, especially as only the truly deep support/engineering area even knew there was such a thing, and could read back and discuss, with reticence you'll understand, logs. Meanwhile there was every question of whether things like reasonably active Ajax were actually breaking their server, which was the appearance.

So if there is to be a worker, it needs to be completely optional, and reasons very visibly documented. A plugin probably has the advantage of being able to make this most clear.

[ostark]

No doubt! @narration-sd I do respect @brandonkelly's decision. But I'm not sure if the "shared hosting" argument is reasonable, since Craft3 requires PHP7, embraces composer and ships with an own cli. Obviously, I'm biased ;-)

[narration-sd]

ha :) I thought I recognized that name....from a very early NA trial, you and Frank.

Understand your view, but decent hosting e.g. ASmallOrange has php7/composer/ssh access, even if we mostly use droplets/PAAS, and yes, I think you are the best there. Composer will be behind the scenes on C3 release as far as I know also.

A plugin is sure to contribute...and now I have broken my no-more-Euro hours pledge anyway over something else, so will sleep...take care

[ostark]

Here we go (v1.1.0)
https://github.com/ostark/craft-async-queue

[timkelty]

I've been using a dedicated container running craft queue/listen, with runQueueAutomatically set to false…

With that setup, is the only advantage using craft-async-queue, that the jobs would always get executed immediately, or is there something else I'm missing?

[ostark]

@timkelty craft-async-queue is a bit more economic compared to a supervisored background process (craft queue/listen) that constantly asks the db for new jobs.

[timkelty]

@ostark makes sense - I'm getting around this (I think?) by using \yii\queue\redis\Queue for my queue instead of the db.

[ostark]

I never tried an alternative queue backend with Craft.

On the PHP side:
Not sure how mature Yii's queue/listen is. Laravel's queue:listen / queue:work became more efficent in ^5.4 in terms of resource usage.

[khalwat]

I still would love to see something like this in core. The current system of pinging a controller that just max's out the system's memory_limit and max_execution_time seems less elegant than running the queue job as a background PHP process.

[timkelty]

@khalwat same here. I don't like relying on plugin for something as important as running jobs. (which is why I don't, I just run craft queue/listen as a daemonized process with runQueueAutomatically=false)