Magento2: Magento crun runs more than once, eventually crashing server

[craigcarnell]

Preconditions

1. Magento 2.2.4 CE and EE
2. PHP 7.1.7
3. MySQL 5.7.19
4. Amazon EC2 Instance - C5.xlarge, Ubuntu 16.04.4
5. Amazon RDS Instance - t2.small

Steps to reproduce

First of all there are about 5 magento installations on this server..

1. Setup the cron to run from cron.d according to instructions given
2. Wait for the cron to execute
3. If the current cron has not finished executing, another instance of the cron is run.
4. This creates even more crons..
5. Server runs out of ram, shutting down php and redis
6. Server stops responding and crashes.

Expected result

1. In Magento 1.x there was a cron.sh that checks to see if the cron is running. Perhaps this should be re-added back to Magento 2?

Actual result

1. Server crashes.

[hostep]

Join the club ;)
We've seen it happen as well on servers with multiple Magento installations.

Currently we are fixing it by prepending the cron command with the flock command which makes sure only a single instance of the same cron can run based on a lock file. We are currently using something like this:

* * * * * flock -w 1 /path/to/some/lock/file.lock php /path/to/magento/installation/bin/magento cron:run

[craigcarnell]

@hostep Thanks, your a life saver! Magento.. get on this!

[hostep]

Be aware that you still might run into more cron job problems. This won't fix everything. I still see cases where the cronjob status in the database is still at running even though the cronjob no longer runs (for example due to the jobs getting killed because the server ran out of memory).
Which then in turn causes a whole bunch of new jobs being inserted with status pending and they never get handled and only pile up forever, until someone goes manually in the database and cleans up those supposedly "running" jobs.
But that's being handled in https://github.com/magento/magento2/issues/11002 (even though there is no solution as far as I'm aware)

[hostep]

Just a quick update here: for some reason I can not understand yet, it looks like when using the flock solution, that parallel execution of cronjobs doesn't work properly anymore, which is kinda strange, since the main cronjob spawns other cron processes in the background which aren't being locked by flock, so I don't quite understand what is happening there.
(this comment is based on something I saw a month or two ago, using Magento 2.1.11 if I remember correctly)

[erikhansen]

For anyone that stumbles upon this old issue, Magento now has built-in locking for CRON processes. Details can be found here: https://devdocs.magento.com/guides/v2.3/install-gde/install/cli/install-cli-subcommands-lock.html