Ddev: Add cron support inside the web container

[mfrieling]

Is your feature request related to a problem? Please describe.
To test correct execution of cron tasks (e. g. drush cron, drupal cron:execute or packages/Symfony bundles depending on mtdowling/cron-expression, inculding laravel/framework ) it would be really helpful or in case DDEV is used for staging environments.

Describe the solution you'd like
Be able to run cronjobs via crontab -e inside the web container, e. g. using a file .ddev/cron which is loaded into crontab automatically on ddev start if it exists.

Describe alternatives you've considered
The problem with using crontab from the host (I tested this only on Debian) is that the environment is completely different than if you execute the task manually. While it works fine manually executed by crontab it could fail due to missing stuff. The first error I got was that DDEV cannot be found. Calling it with the full path to the executable or loading ~/.profile before running ddev exec didn't help as that produced other errors, like

Failed to create required directory /home/.ddev, err: mkdir /home/.ddev: permission denied

or

docker-compose does not appear to be installed.

[rfay]

I don't think crond is possible (at least easily), because it requires installing and running crond in the container.

Note that the problem you had with not being able to create /home/.ddev should not happen any more in v1.10+ (prerelease today), at least I'd love to have you try it. Each user gets a real installed user now, with a home directory with the right perms, etc.

Any solution that does not require having crond running in the container is a possible win.

[mfrieling]

Understand. A solution running on the host which is capable of executing anything via ddev exec which can be executed manually this way should be sufficient too.

Maybe the mentioned .ddev/cron file which either just contains the (container-side) command to run via ddev exec or with a cron expression, in case something like mtdowling/cron-expression exists for the language DDEV is developed with. Then either utilize native crond on the host if available or just run a background process which triggers in a defined interval (e. g. every minute) for each project which has cron enabled. The best would be to introduce a parameter similar to xdebug_enabled in config.yaml

cron_enabled: false|true|[time-interval]

• false: don't execute cron for this project
• true: execute cron for this project with the default (or maybe globally configured) time interval
• [time-interval]: something like 1m (every minute), 5m (every five minutes), 1h (every hour), ...

[thomaskieslich]

Hi, I think have a Solution without change ddev. I will describe how it works, in my Case with TYPO3.
Step 1
Create a file 'typo3' inside a new created 'corn' Folder inside .ddev.
# m h dom mon dow user command */1 * * * * root TYPO3_CONTEXT=Development/Local /usr/bin/php /var/www/html/public/typo3/sysext/core/bin/typo3 scheduler:run

i have a docker-compose.override.yml with this content, the important Line is the last one:
`version: '3.6'

services:
web:
environment:
- TYPO3_CONTEXT=Development/Local
volumes:
# mount composer cache
- ~/.composer/cache:/home/application/.composer/cache
# add php xdebug profile path
- ./data/php-profiles/:/tmp/debug/:delegated
# add crontab
- ./cron/typo3:/etc/cron.d/typo3`
to link the cron file to the container.

To install the install cron into the Container i add this hooks to config.yml:
hooks: post-start: - exec: sudo apt-get -y update - exec: sudo chmod 0600 /etc/cron.d/typo3 - exec: sudo apt-get -y install --no-install-recommends cron - exec: sudo apt-get -y clean - exec: sudo service cron start

It works with root because the php file can exec from all users.
Repo: https://github.com/thomaskieslich/ddev-typo3/tree/development/.ddev

I hope it works for you and perhaps this can be added to the docs ;-)

[rfay]

Wow, that's great, thanks. I imagine that adds lots of stuff, but if it's what anybody needs it's fine.

I'd recommend doing the package in the config.yaml, much, much easier: webimage_extra_packages: ["cron"], and you can probably do everything else you might want in .ddev/web-build/Dockerfile too... or maybe add the package there (same result).

Would appreciate if you could expand on this and do a PR to github.com/drud/ddev-contrib ! Thanks!

[thomaskieslich]

With --no-install-recommends it needs 247kb on Disk ;-) I will see how i can realize it with webimage_extra_packages and then i can make a PR.
Thanks for the explanation of the Possibilities.

[davdenic]

I have errors in both ways, log says:

Setting up exim4-config (4.89-2+deb9u5) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
Adding system-user for exim (v4)
adduser: No UID/GID pair is available in the range 100-999 (FIRST_SYS_UID - LAST_SYS_UID).
adduser: The user `Debian-exim' was not created.
dpkg: error processing package exim4-config (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of exim4-base:
exim4-base depends on exim4-config (>= 4.82) | exim4-config-2; however:
Package exim4-config is not configured yet.
Package exim4-config-2 is not installed.
Package exim4-config which provides exim4-config-2 is not configured yet.

dpkg: error processing package exim4-base (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of exim4:
exim4 depends on exim4-base (>= 4.89-2+deb9u5); however:
Package exim4-base is not configured yet.
exim4 depends on exim4-base (<< 4.89-2+deb9u5.1); however:
Package exim4-base is not configured yet.

dpkg: error processing package exim4 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of exim4-daemon-light:
exim4-daemon-light depends on exim4-base (>= 4.89); however:
Package exim4-base is not configured yet.

dpkg: error processing package exim4-daemon-light (--configure):
dependency problems - leaving unconfigured
Setting up cron (3.0pl1-128+deb9u1) ...
addgroup: No GID is available in the range 100-999 (FIRST_SYS_GID - LAST_SYS_GID).
addgroup: The group `crontab' was not created.
dpkg: error processing package cron (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mailutils:
mailutils depends on default-mta | mail-transport-agent; however:
Package default-mta is not installed.
Package exim4-daemon-light which provides default-mta is not configured yet.
Package mail-transport-agent is not installed.
Package exim4-daemon-light which provides mail-transport-agent is not configured yet.

dpkg: error processing package mailutils (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
exim4-config
exim4-base
exim4
exim4-daemon-light
cron
mailutils
E: Sub-process /usr/bin/dpkg returned an error code (1)
', stderr='Building db
Building web
Service 'web' failed to build: The command '/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y cron' returned a non-zero code: 100'

[thomaskieslich]

you must use the --no-install-recommends Option.
apt-get -y install --no-install-recommends cron

[rfay]

Since that is in fact the correct approach, I added it to the implementation of *image_extra_packages in https://github.com/drud/ddev/pull/1776

[rfay]

BTW, I tested this technique and it works great. The magic is in starting cron in the post-start hook, which is what I hadn't considered before. I had always hesitated because systemctl or whatever has to start all the services... but here you are starting it manually. Nice work!

[thomaskieslich]

Thanks, i am new with ddev since Version 1.10 (because it has cmmands) and the TYPO3 Developer Days. Thanks for your nice Job. I will try out different Ways to run cron and then i will make a PR to ddev-contrib Repo. My current Favorit is to run cron with a Command File in commands/web because i can only Start it if i need it.
commands/web/cron
sudo apt-get -y update sudo apt-get -y install --no-install-recommends cron sudo apt-get -y clean sudo chmod 0600 /etc/cron.d/typo3 sudo service cron start

[rfay]

https://github.com/drud/ddev-contrib/blob/master/recipes/cronjob/README.md now has a nice recipe for cron. Thanks @thomaskieslich .

[mieserfettsack]

Maybe you can just install cron inside the web container. I use real cron inside webs development container to have a identical setup to live.

Add cron to web container by creating or eddting this file .ddev/web-build/Dockerfile

ARG BASE_IMAGE=drud/ddev-webserver:v1.11.0
FROM $BASE_IMAGE

# I use dos2unix to make sure that no CRLF file file be executed, which will cause errors
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends --no-install-suggests install -y dos2unix
# In older versions of ddev it was not possible to install cron without adding group crontab to /etc/group manualy
# There exist hundreds of groups. The following sed replaced on of them to group crontab.
RUN sed -i "s/gid_1337/crontab/" /etc/group
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends --no-install-suggests install -y cron

Create a file which will be imported to crontab .ddev/Config/cron/crontab.dev

*/1 * * * * /bin/ls > /var/www/ls_output
*/1 * * * * /bin/uname -a > /var/www/uname_output

To start your crontab.dev file as cron inside web-container you can use post-start-hooks. Add the following lines to .ddev/config.yaml

This will import you crontab.dev to webs crontab and will restart cron afterwards

hooks:
  post-start:
    - exec: /bin/cat ./.ddev/Config/cron/crontab.dev | /usr/bin/dos2unix | /usr/bin/sudo crontab -
    - exec: /usr/bin/sudo /etc/init.d/cron restart

Antoher way to achieve this is a command like ddev prepareEnvironment. Create the file .ddev/commands/web/prepareEnvironment and add the lines as shown:

#!/bin/bash

echo "Running prepareWebContainer"
# some other usefull commands
/bin/cat ./.ddev/Config/cron/crontab.dev | /usr/bin/dos2unix | /usr/bin/sudo crontab -
/usr/bin/sudo /etc/init.d/cron restart
# more awesome commands

[bfuzze]

Both of these approaches seem to work...

• First approach is typo3 specific, whereas I need a framework agnostic solution.
• Second approach modifies the Docker build which seems unnecessary and I prefer to avoid if possible.

I settled on the following composite solution...

.ddev/config.yaml

webimage_extra_packages: ["cron --no-install-recommends", "dos2unix --no-install-recommends"]

hooks:
  post-start:
    - exec: /bin/cat /mnt/ddev_config/crontab | /usr/bin/dos2unix | crontab -
    - exec: /usr/bin/sudo /etc/init.d/cron restart

.ddev/crontab

*/1 * * * * echo "CHECK2" >> /var/www/html/temp/cron.log

.ddev/commands/web/reload-crontab
(so we can adjust crontab with restarting the container)

#!/usr/bin/env bash

## Description: Reloads cron from .ddev/crontab
## Usage: reload-crontab
## Example: "ddev reload-crontab"

/bin/cat /mnt/ddev_config/crontab | /usr/bin/dos2unix | crontab -

Thanks.

UPDATE:
For future readers, I ran into a lot of issues running calling drush through crontab (Drupal 8.9) with this setup. For some reason, Drush only loads the settings.php (does not pull in the settings.ddev.php), which is quite strange. I had to transfer ddev settings over to get it to go. I also suggest using non-root user crontab

[andrewbelcher]

FYI for those having issues with drush and loading settings.ddev.php, it is likely because the include checks for the IS_DDEV_PROJECT environment variable. You can provide that by setting up your crontab like:

IS_DDEV_PROJECT=true
*/5 * * * * /var/www/html/vendor/bin/drush --root=/var/www/html/web core-cron -l https://example.ddev.site >> /tmp/cron.log 2>&1

And for reference for those who need to do this along side a customer Dockerfile (as I did as I install some other built tools), instead of using webimage_extra_packages you can use a .ddev/web-build/Dockerfile containing:

ARG BASE_IMAGE
FROM $BASE_IMAGE

RUN apt-get update

# Install cron.
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends --no-install-suggests cron

# Do whatever else you need to do...

I didn't bother with dos2unix as, but you can add that in the same manner.

Edit

You may find that still fails if you use other environmental variables injected into the container, as they also will not be available. A more complete solution would be to do something like:

_reload-crontab_:

{ env ; cat /mnt/ddev_config/crontab ; } | crontab -

_config.yaml_:

hooks:
  post-start:
  - exec: "{ env ; cat /mnt/ddev_config/crontab ; } | crontab -"
  - exec: sudo /etc/init.d/cron restart

This essentially imports .ddev/crontab but prepends all the env variables injected into the container.