Caddy: Infinite Redirect with Caddy 0.9.3

[dominikschulz]

1. What version of Caddy are you running (caddy -version)?

Caddy 0.9.3

2. What are you trying to do?

Serve https with automatic SSL

3. What is your entire Caddyfile?

https://sub.domain.com

log stdout
prometheus 0.0.0.0:9180

tls [email protected]

proxy / upstream:80 {
  transparent
}

4. How did you run Caddy (give the full command and describe the execution environment)?

docker run -d --link upstream -p 0.0.0.0:80:80 -p 0.0.0.0:443:443 my-caddy-image

5. What did you expect to see?

The content of https://sub.domain.com/

6. What did you see instead (give full error messages and/or log)?

Infinite redirects from https://sub.domain.com/ to https://sub.domain.com/

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

Use this Dockerfile:

FROM alpine:3.4
RUN apk --update --no-cache add \
  ca-certificates \
  curl \
  git \
  openssh-client \
  && rm -rf /var/cache/apk/*
RUN mkdir -p /bin /srv
RUN curl -fsSL "https://caddyserver.com/download/build?os=linux&arch=amd64&features=cors%2Cgit%2Chugo%2Cminify%2Cprometheus%2Cratelimit%2Crealip%2Csearch%2Ccloudflare%2Cgooglecloud%2Croute53" -o /tmp/caddy.tar.gz \
  && cd /tmp \
  && tar xvzf caddy.tar.gz \
  && mv /tmp/caddy /bin/caddy \
  && rm -rf /tmp/caddy.tar.gz
VOLUME [ "/srv" ]
WORKDIR /srv
ADD Caddyfile /Caddyfile
EXPOSE 80 443 9180
CMD [ "/bin/caddy", "-http2=false" ]

and the above Caddyfile, run:

docker build -t my-caddy-image .
docker run -d -p 0.0.0.0:80:80 -p 0.0.0.0:443:443 my-caddy-image

[k0nsl]

I "solved" it by changing the caddyfile to:

sub.domain.com

log stdout
prometheus 0.0.0.0:9180

tls [email protected]

proxy / upstream:80 {
  transparent
}

Try that, @dominikschulz.

[elcore]

Hello @dominikschulz and @k0nsl,

I am not able to reproduce your issue, please clear your cache and try again.

TESTED WITH CHROME, FIREFOX, CURL

Caddyfile:

https://domain.com {
   ............
}

Custom Caddy:

ISSUE 1152: HOST: domain.com
ISSUE 1152: RedirTo: https://domain.com/

ping @mholt

[k0nsl]

This was only an issue for me when I specified https://domain.tld in my caddyfile. When I simply specified domain.tld the redirect loop was solved.

I am using the latest version and the web server is acting as a reverse proxy and nothing more.

[elcore]

@k0nsl I am using https://domain.tld (just for the test) 😄

[abiosoft]

Can't reproduce it either. What is upstream:80 doing ?

[mholt]

I'm suspicious of the upstream...

[twmbx]

Is this not similar to the problems we were discussing in this forum post?

[tw4452852]

Maybe the redirection comes from the auto https (it adds the redirection internally).

[mholt]

It adds a redirect from HTTP to HTTPS, but not HTTPS to HTTPS. :confused: So I'm still suspicious of the upstream, since I can't reproduce this.

[elcore]

Hello @dominikschulz.

try this:

https://sub.domain.com

log stdout
prometheus 0.0.0.0:9180

tls [email protected]

proxy / https://upstream {
  transparent
}

It adds a redirect from HTTP to HTTPS, but not HTTPS to HTTPS. 😕 So I'm still suspicious of the upstream, since I can't reproduce this.

Me too @mholt

[arraisgabriel]

@elcore I've had the same problem here and your suggestion fixed it.
In my case, my upstream was redirecting the request to HTTPS, causing the infinite loop.

Thank you =)

[elcore]

@arraisgabriel 👍, have fun with Caddy!

[dominikschulz]

I'll double check the upstream (GitLab in our case). I was guessing at Caddy because it works flawlessly with any previous version of Caddy.

[mholt]

If you take the proxy directive out, does it still infinitely redirect? If not, I'm even more suspicious of the upstream app.

(Closing to clean up our todo list unless this can be verified as a bug in Caddy -- feel free to reopen if that's the case!)

[dominikschulz]

Upstream responds properly:

curl -H'Host: gitlab.example.org' -i http://upstream:80/ HTTP/1.1 302 Found Server: nginx Date: Fri, 09 Dec 2016 16:51:45 GMT Content-Type: text/html; charset=utf-8 Content-Length: 108 Connection: keep-alive Cache-Control: no-cache Location: https://gitlab.example.org/users/sign_in [...] Strict-Transport-Security: max-age=2592000;

Bug is still present in latest caddy release.

[metalmatze]

I quickly want to add, that we also tried the Caddyfile without the https:// prefix.
Unfortunately the same problem occurs when proxying to drone. So it's definitely a Caddy problem and not a upstream one... 😕

[mholt]

@dominikschulz @metalmatze No, I'm still suspicious of the upstream. We've seen this before when web apps think they are in charge of redirecting to HTTPS. Be very careful how you're proxying and ensure your app isn't overstepping its bounds and being a bit too cavalier with redirecting.

[metalmatze]

It's just basic gitlab. We run exactly sameersbn/gitlab as container and proxy with caddy to it. Pretty default configurations:

GITLAB_BACKUPS: 'daily'
GITLAB_BACKUP_TIME: '01:00'
GITLAB_BACKUP_EXPIRY: '172800'
GITLAB_EMAIL: '[email protected]'
GITLAB_EMAIL_REPLY_TO: '[email protected]'
GITLAB_HOST: 'gitlab.example.com'
GITLAB_HTTPS: 'true'
GITLAB_HTTPS_HSTS_MAXAGE: '2592000'
GITLAB_INCOMING_EMAIL_ADDRESS: '[email protected]'
GITLAB_INCOMING_EMAIL_ENABLED: 'false'
GITLAB_NOTIFY_ON_BROKEN_BUILDS: 'true'
GITLAB_NOTIFY_PUSHER: 'false'
GITLAB_PORT: '443'
GITLAB_SECRETS_DB_KEY_BASE: 'XXX'
GITLAB_SECRETS_SECRET_KEY_BASE: 'XXX'
GITLAB_SECRETS_OTP_KEY_BASE: 'XXX'
GITLAB_SSH_PORT: '22'
GITLAB_TIMEZONE: 'Berlin'

Everything else should be uninteresting because it's just email configurations.

[mholt]

So if you're proxying to GitLab via HTTP and it expects HTTPS as you config shows, that could be the reason for the redirect.

[zwopir]

this was neither a bug in the upstream nor caddy, but a unlucky combination of configurations:

• the stack was running in docker-compose
• the upstream in the caddyfile was set to gitlab (plain no https scheme, so implicitly http)
• the docker-compose service for gitlab was linked to the name gitlab to allow caddy to talk to the gitlab container
• on a non-https request, caddy redirects to https://gitlab.domain.com
• on a https request, the upstream name gitlab is resolved to gitlab.domain.com and a request to caddy (_not_ the upstream) is issued - triggering the https redirect, since the backend is configured as plain http.

As a lesson learned: don't name your upstream to something that can be resolved to the fqdn caddy is listening on. Maybe this is worth mentioning in the docs, @mholt ?

[mholt]

Wow, thanks for following up! That's very interesting.

Uhm, I guess we could consider it, but IMO that's kind of obvious (even though I understand how that kind of detail can get lost in the complex setup with docker-compose, etc). I mean, ultimately, you have to make sure the DNS is setup correctly if you're using hostnames instead of IP addresses to reverse proxy...

[zwopir]

retrospectively, yes - it's obvious. I was just wondering if anybody is trying to run caddy with a upstream in a similar stack. But googling the problem will hopefully reveal this issue ...