Ddev: Create a service configuration for varnish

[tannerjfco]

What happened (or feature request):

Feature Request
In #121 we determined varnish was one of several popular services that can be a requirement for more complex applications. In order for ddev to facilitate local environments for these more complex applications, we should provide an optional service configuration for varnish.

What you expected to happen:

A docker-compose configuration should be made, named docker-compose.varnish.yml. This configuration should:

• Determine if an existing varnish docker image can be used, or if we need to develop / maintain our own.
• Define a service named varnish that provides a varnish container.
• Where possible, pre-configure the redis container to reduce required setup for a user.
• Document any steps that are necessary after adding the service file to .ddev to be able to use varnish with a Drupal site.

The example solr service in documentation is a good reference for how this service file should work.

How to reproduce it (as minimally and precisely as possible):

Anything else do we need to know:

Related source links or issues:

[beeradb]

Varnish is an interesting one. My initial reaction is that it would be ideal for varnish to take over port 80, and the web container would listen on an alternative port.

That being said, it might not be the end of the world if varnish was exposed on a non-default port. It sure would be nice to ensure these additional services get surfaced via ddev describe output, though. I wonder if we could expose that info via labels?

[rickmanelius]

Closing for now until this becomes an end-user request, at which time we can re-open.

[jeffsheltren]

+1 on adding Varnish support

[dclear]

re-opening :)

[rfay]

@jeffsheltren do you have thoughts on how varnish could best be done? Separate container? Localhost port binding only? Would the ddev-router front varnish, which would front the web container?

[jeffsheltren]

Putting Varnish in its own container makes sense to me. Are you currently terminating SSL in ddev-router? If so, it should be pretty easy to put Varnish in between router and web containers.

It would be nice to leave an option to disable/bypass Varnish completely. And you'll need to allow VCL config to be supplied by the user.

[rfay]

Yes, the ddev-router typically terminates SSL, although https is exposed in addition on the localhost https port.

Thanks!

[yareckon]

Hi, this is how we are doing this at wegewerk:

# DDev Varnish recipe file.
#
# To use this in your own project:
# 1. Copy this file to your project's ".ddev" directory.
# 2. Create the folder path ".ddev/varnish/".
# 3. Make a file with the following content at
#    ".ddev/varnish/default.vcl".
#
# vcl 4.0;
#
# backend default {
#   .host = "web:80";
# }
#
# Notes:
# - The configuration files must be present before running "ddev start".
# - Changes to the configuration files will not be automatically applied.
#
# To access the site through Varnish after it is installed simply visit the ddev
# site as usual. To avoid going through varnish visit the the novarnish. subdomain.

version: '3.6'

services:
  # This is the service name used when running ddev commands accepting the
  # --service flag.
  varnish:
    # This is the name of the container. It is recommended to follow the same
    # name convention used in the main docker-compose.yml file.
    container_name: ddev-${DDEV_SITENAME}-varnish
    image: varnish:6.2
    # These labels ensure this service is discoverable by ddev.
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: $DDEV_APPROOT
    environment:
      # This defines the host name the service should be accessible from. This
      # will be sitename.ddev.site.
      # This is the first half of the trick that puts varnish "in front of" the
      # web container, just by switching the names
      - VIRTUAL_HOST=$DDEV_HOSTNAME
      # This defines the ports the service should be accessible from at
      # sitename.ddev.site.
      - HTTP_EXPOSE=80:80,443:443
      - HTTPS_EXPOSE=443:443
    volumes:
      # This exposes a mount to the host system `.ddev/varnish` directory where
      # your default.vcl should be
      - ./varnish:/etc/varnish
      - ".:/mnt/ddev_config"
    ports: []
    links:
      - web:web
  # This is the second half of the trick that puts varnish "in front of" the web
  # container, just by switching the names
  web:
    environment:
      - VIRTUAL_HOST=novarnish.$DDEV_HOSTNAME

[briantully]

Thanks for the above snippet @yareckon ! For some reason I was able to use your config and put it in a docker-compose.varnish.yaml and got it working a few months back on a different project.

However on a new project when I include the docker-compose.varnish.yaml, I get the following error:

503: No ddev back-end site available.

This is the ddev-router container: There is no back-end webserver at the URL you specified. You may want to use "ddev start" to start the site.

I'm not sure if it's due to a newer version of ddev or if your config needs to be updated to contain unique values per project.

Any ideas?

[briantully]

DOH. I'm guessing it's because I don't have a default VCL. Hmmmm... so maybe I never did get it working.

Is there a recommended default.vcl anyone can share for DDEV and Drupal?

[rfay]

He suggests it above:

 # 3. Make a file with the following content at
#    ".ddev/varnish/default.vcl".
#
# vcl 4.0;
#
# backend default {
#   .host = "web:80";
# }

[briantully]

SONOFA. Haha. I'll just go stand in the corner and face the wall... ;)

[rfay]

Your penance is to write this up as a PR for github.com/drud/ddev-contrib once you've got it all sorted out :)

[briantully]

Thank you, Father Randy. My penance is humbly accepted ;)

[rikwillems]

I found these lines to conflict with additional_hostnames and additional_fqdns settings.

      - HTTP_EXPOSE=80:80,443:443
      - HTTPS_EXPOSE=443:443

Changing it in the following makes it run nice:

      - HTTP_EXPOSE=80:8080,4443:4443
      - HTTPS_EXPOSE=443:4443

[briantully]

@rikwillems any chance you can share your whole process of getting varnish working in DDEV? I'm still getting a 503 no back-end site available, following all the steps above and including your recent changes.

I've also tried ddev ssh --service varnish but I get the following error:

ERROR: No container found for varnish_1

[rikwillems]

@briantully all I really did was adding these files. The docker-compose.varnish.yaml file is the on above with my mentioned changes. And then restart your ddev project.

# .ddev/docker-compose.varnish.yaml
version: '3.6'

services:
    # This is the service name used when running ddev commands accepting the
    # --service flag.
    varnish:
        # This is the name of the container. It is recommended to follow the same
        # name convention used in the main docker-compose.yml file.
        container_name: ddev-${DDEV_SITENAME}-varnish
        image: varnish:6.2
        # These labels ensure this service is discoverable by ddev.
        labels:
            com.ddev.site-name: ${DDEV_SITENAME}
            com.ddev.approot: $DDEV_APPROOT
        environment:
            # This defines the host name the service should be accessible from. This
            # will be sitename.ddev.site.
            # This is the first half of the trick that puts varnish "in front of" the
            # web container, just by switching the names
            - VIRTUAL_HOST=$DDEV_HOSTNAME
            # This defines the ports the service should be accessible from at
            # sitename.ddev.site.
            - HTTP_EXPOSE=80:8080,443:4443
            - HTTPS_EXPOSE=443:4443
        volumes:
            # This exposes a mount to the host system `.ddev/varnish` directory where
            # your default.vcl should be
            - ./varnish:/etc/varnish
            - ".:/mnt/ddev_config"
        ports: []
        links:
            - web:web
    # This is the second half of the trick that puts varnish "in front of" the web
    # container, just by switching the names
    web:
        environment:
            - VIRTUAL_HOST=novarnish.$DDEV_HOSTNAME

# .ddev/varnish/default.vcl
vcl 4.0;

backend default {
    .host = "web";
    .port = "80";
}

acl purge {
    "web";
}

[rfay]

@rikwillems I'd love it if you could do a PR for https://github.com/drud/ddev-contrib that explained this in detail and made it more accessible for people. Thanks!

[stborchert]

Tried this on a new project and also get "503: No ddev back-end site available." for the default domain (while "novarnish.{domain}" works without any problems).
ddev ssh -s varnish also fails with the know error "No container found for varnish_1".

As a rough guess I would say the vcl is not loaded ... unfortunately ddev logs doesn't say anything about varnish.
@rikwillems any ideas what could be wrong?

I also tried using the latest version of the varnish docker image and directly mount the _default.vcl_ but that didn't work either.

[rfay]

You'd use ddev logs -s varnish to see the varnish logs, or perhaps ddev ssh -s varnish and inspect the container looking for logs.

If you're getting "No container found for varnish_1" that means that the varnish container isn't running. You can docker ps to see running containers, and you can see docker ps -a to see running and stopped containers. Most likely your varnish container is exiting on startup, perhaps due to an error in the default.vcl?. ddev logs -s varnish may help with that, or docker logs ddev-<projectname>-varnish

[stborchert]

Bingo, its an error with the vcl:

Error:
Message from VCC-compiler:
Unused acl purge, defined:
('/etc/varnish/default.vcl' Line 9 Pos 5)
acl purge {
----#####--

Running VCC-compiler failed, exited with 2
VCL compilation failed

Seems I have to use some docs to "learn" vcl syntax :)

[rfay]

Still looking for one of you to write this up in https://github.com/drud/ddev-contrib :) (or somewhere else like Stack Overflow if that's easier for you). But a full working solution that can be maintained will really help people.

[stborchert]

Uh, that was easy: "Unused acl purge" ...
It works if I remove the acl so the _default.vcl_ looks like this:

vcl 4.1;

backend default {
  .host = "web";
  .port = "80";
}

Ok, so now we have to add a real-world-vcl and everything will be fine.

Thanks for your work!

[stborchert]

Still looking for one of you to write this up in https://github.com/drud/ddev-contrib :)

Will do in a minute.

[rikwillems]

Thank you @stborchert for writing it down like this.