Caddy: It seems caddy proxy have a dns cache.

[hyacinthus]

I want to use caddy as a API gateway in docker swarm, but It seems caddy (or some other package) will cache the proxy destination dns lookup resault for several minutes.

1. What version of Caddy are you running (caddy -version)?

0.9.5

2. What are you trying to do?

This is my docker-compose file:

version: '3'
services:
    app:
        image: muninn/caddy-microservice:app
        ports:
            - 12345:12345
        deploy:
            replicas: 3
    gateway:
        image: muninn/caddy-microservice:gateway
        ports:
            - 2015:2015
        depends_on:
            - app
        deploy:
            replicas: 1
            placement:
                constraints: [node.role == manager]

I use "docker stack deploy -c docker-compose.yml caddy" to start it,
and check if caddy can route http request to a random backend app container.

3. What is your entire Caddyfile?

:2015 {
    proxy / app:12345
}

4. How did you run Caddy (give the full command and describe the execution environment)?

Dockerfile:

FROM alpine
MAINTAINER Muninn <[email protected]>

LABEL caddy_version="0.9.5" architecture="amd64"

ARG plugins=cors

RUN apk add --no-cache openssh-client git tar curl

RUN curl --silent --show-error --fail --location \
      --header "Accept: application/tar+gzip, application/x-gzip, application/octet-stream" -o - \
      "https://caddyserver.com/download/build?os=linux&arch=amd64&features=${plugins}" \
    | tar --no-same-owner -C /usr/bin/ -xz caddy \
 && chmod 0755 /usr/bin/caddy \
 && /usr/bin/caddy -version

EXPOSE 80 443 2015
VOLUME /root/.caddy
WORKDIR /www

COPY Caddyfile /etc/Caddyfile

ENTRYPOINT ["/usr/bin/caddy"]
CMD ["--conf", "/etc/Caddyfile", "--log", "stdout"]

5. What did you expect to see?

curl http://my-host-ip:2015

I expect that caddy can balance requests to all backend apps.

6. What did you see instead (give full error messages and/or log)?

1. When I "curl http://my-host-ip:12345" on my host, docker swarm will route my request to a random app container.
2. When I "curl http://app:12345" in gateway container, will still arrive a random container.
3. When I "curl http://my-host-ip:2015" on my host, caddy just route me to same container every time.
4. But! several minutes later , I arrived another container from caddy, and only this container in next few minutes. So I say perhaps caddy cached the proxy destination dns lookup resault for several minutes.

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

I published my demo to docker cloud, so you can just clone the demo and run
or just copy the compose file above and run

docker-compose pull
docker stack deploy -c docker-compose.yml caddy

oh, you need to have docker engine 1.13 higher and switch it to swarm mode.

[mholt]

What if you set keepalive 0 inside proxy and try again?

[hyacinthus]

@mholt Set keepalive 0 works, I'll try concurrent requests to caddy when keepalive is set to default. Perhaps everything goes well.

[mholt]

If that's the case, I suspect that is just Caddy reusing the connection it first opened, so no DNS lookups are being performed anyway; if this is true, it has nothing to do with DNS and is just a keepalive tweak to make in your config. Let me know if you still experience problems with keepalives disabled!

[hyacinthus]

Thank you for your great work.

I tested it with a large number concurrent requests just now. All went well.

I'll use caddy instead of nginx in my project.