Freescout: Cron job gets killed: 'queue:work' not working

[imc67]

Hi,

I discovered that queue:work is sometimes working and sometimes not (system > status > Cron command = queue:work | Laatste run: ? Laatste succesvolle run: ? Probeer de cache te legen om de start van het commando te forceren.)

Steps to reproduce:

1. flush cache
2. it's working (system > status > Cron command = queue:work = green) and jobs are processed
3. after a while I get a Cron error emailed from the server:

/bin/sh: line 1: 521495 Killed /usr/local/bin/php /home/abcd/domains/abcd.tv/public_html/redactie/artisan schedule:run > /dev/null 2>&1

What could this be and how can I solve this?

[imc67]

@freescout-helpdesk do you have any idea?

Situation:

1. We use DirectAdmin on the server.
2. PHP version is 7.3.12.
3. Disabled (default) PHP settings: system,exec,passthru,proc_nice,proc_terminate,dl,popen,pcntl_exec
4. CRON is working (freescout:fetch-emails)
5. No errors in the logfiles

If I don't flush the cache and wait for the queue:work to run there are no auto-reply mails etc. send.

I introduced FreeScout in our, small volunteer, organization and everybody was so excited to use it that we put it in production today. However manually flushing cache to get mails send is not nice.

What do I have to do to make it work?

Kind regards,

Marcel.

[freescout-helpdesk]

You need to debug https://github.com/freescout-helpdesk/freescout/blob/dist/app/Console/Kernel.php

[imc67]

I tried to figure out that file, I'm not a programmer but think I noticed that the required PHP settings are sufficient at least.

In your wiki I found some info for debugging but I can't enter that server via SSH and in the log files are no errors.

I'm a bit lost after 3 days of trying to find the reason and manually flushing cache.

[freescout-helpdesk]

Most likely something is killing queue:work command after it starts (maybe some operating system max execution time limitation). Are you on a shared hosting?

So you need an admin who can find out why the command is being killed or contact your hosting provider.

[imc67]

Hi,

I asked my provider what could be wrong. In their logfiles they can't see a reason for the killed job. The Max_Excution_Time is already set to max (900).

They think it might be because the cron is called every minute and maybe the job is not finished before the next call?

The strange thing is that when I clear cache it works but after minimal one hour (but sometimes longer) it gets killed. How to check if it's a specific sub-job?

For now I set the Cron job to every 2 minutes, just to check if it helps....

[imc67]

after 16 hours and many jobs getting killed it's still a big issue :-(

[freescout-helpdesk]

No use to spam here. Find a developer.

[imc67]

well well, that's what you call "customer service" ........

[freescout-helpdesk]

If you are a developer, you can fix the issue and create a pull request (see Development Guide).

I you are not a developer, you can find a developer who will diagnose the issue and you can create a detailed bug report and community will try to fix the issue. Here is the good example https://github.com/freescout-helpdesk/freescout/issues/387

This is how it works in open source.

[nebulade]

We seem to hit this as well with the Cloudron package (on php 7.2 though). However the stalling of the queue:work is not happening always. I am currently debugging this, trying to understand what this task actually does. In our case it seems it is triggered by the regular php artisan schedule:run and thus causing that to never exit.

Any further information on this would be much appreciated, since I am not really an artisan/Laravel expert and -vvv does not seem to increase verbosity by much, so this may take some time :-)

[nebulade]

I found the inconsistency around when it gets stuck. The missing detail is that schedule:run not always also schedules queue:work. Not yet sure when it is doing this, but also that seems unrelated now.

Overall this looks more like a laravel topic rather than freescout. From what I can tell we need the --stop-when-empty flag when running in shortlifed Docker containers according to https://laravel.com/docs/5.7/queues#running-the-queue-worker

I have to still figure out why that flag is not available on the package we have here in order to test this.

Thought of posting that here for future reference, but still it does not look like a freescout specific issue, so no need to reopen this issue.

[freescout-helpdesk]

Good research.

queue:work should work in the background continuously. schedule:run just makes sure that queue:work is running and executes it if needed.

[nebulade]

At least for our use-case we have now split up the short- and long-lived tasks, which solves the issue.
Only minor caveat is, that triggering the long-lived queue worker, requires to run it via php artisan schedule:run in order to have the correct queue names, specifically the one with the uuid:

Running scheduled command: '/usr/bin/php7.2' 'artisan' queue:work --queue='emails,default,e4276e7e9795632f945df5bb9b8815c5' --sleep=5 --tries=1

This will also trigger the other tasks, but that does not seem to cause any issue, since the others will be triggered by cron regardless every minute.

Unless there is a better suggestion from the devs, we will keep it at that for the Cloudron app package. Thanks!

[freescout-helpdesk]

@nebulade This solution should work.

[CodeAndWeb]

I am running a worker docker container with a simple for-loop that runs artisan schedule:run with a sleep 60. I run the artisan command synchronous - and was wondering why it only ran once and stalled.

It is surely not the idea of Laravel's task queue to run non-terminating commands.

I'd propose to remove the artisan queue:work from the schedule and update the documentation to run the command separately.

[freescout-helpdesk]

Closing for no activity.