Trellis: Trellis generates a second, self-signed certificate on Staging and Production

[raph-topo]

Description

Trellis generates a second, self-signed certificate for example.com on Staging and Production

Steps to reproduce

Consult the SSL Report of any Trellis-powered server: https://www.ssllabs.com/ssltest/analyze.html?d=roots.io&hideResults=on in section Certificate #2: RSA 2048 bits (SHA256withRSA).

Expected behavior:

One valid certificate for roots.io.

Actual behavior:

Two certificates: roots.io (valid) + example.com (invalid).

Versions

Introduced by https://github.com/roots/trellis/pull/888/commits/90ec6c0f88aeb36c167535ecdea18e0e440e4ac6#diff-6a42c926df0abb8be26f6923b00ee7a5

Additional information

https://github.com/roots/trellis/pull/888#issuecomment-370516859 does state

Implementation notes:
if no sites create a self-signed cert, this creates one for throw-away example.com

But, on Staging and Production, self-signed certs should be very rare.

https://web.archive.org/web/20180819104946/https://tech.mendix.com/linux/2014/10/29/nginx-certs-sni/ seems to explain the need for it; but responding 403 Forbidden when it should probably be 495 SSL Certificate Error or 496 SSL Certificate Required seems odd.

https://github.com/h5bp/server-configs-nginx/pull/177#issuecomment-442769233 makes me think this behavior is outdated, but I am not sure I understand all implications.

Is creating an invalid certificate for an extant domain (example.com is a real domain after all) really the intended behavior?

[swalkinshaw]

🤔 to be honest I had no idea this behaviour existed. I don't entirely understand all of this yet either...

@TangRufus maybe you can help here. You helped reviewed @fullyint's PR from before.

[swalkinshaw]

Okay so here's my conclusion/summary:

• the no-default confs are needed. If you have an SSL enabled site, then we do need the SSL no-default too.
• Nginx requires an SSL cert, otherwise it won't work
• a self-signed cert works fine to satisfy this requirement
• however, you should only care about an SSL no-default server if you have an SSL enabled site. and in that case, we could re-use the existing cert instead of generating this additional self-signed cert

So if we can find an easy way to reference the real cert, we can skip this self-signed one.

I think that's what https://github.com/h5bp/server-configs-nginx/pull/177#issuecomment-442769233 was referencing. Because they restructured their confs to have a reusable certificates.conf file, it can easily be included/re-used. We don't currently generate an equivalent certificate only conf file though.

[raph-topo]

We don't currently generate an equivalent certificate only conf file though.

Ok! The situation makes sense.
So, we first have to solve https://github.com/roots/trellis/issues/1073.

[TangRufus]

If I recall it correctly, before @fullyint's works (#885 & & #888):

Nginx treats the unknown hosts as default_server or the first server block.

After @fullyint's works:

Trellis adds no-default server blocks which drop requests for unknown hosts (i.e: return 444). But, the ssl.no-default server block needs a ssl cert:

Without ssl_certificate and ssl_certificate_key in the default server block, https sites fail:

in browser: "The connection was reset" ERR_CONNECTION_RESET
curl: curl: (35) Server aborted the SSL handshake

Thus, #888 implements

1. if no sites create a self-signed cert, this creates one for throw-away example.com

2. ssl.no-default.conf uses the first existing self-signed cert, if any, else the example.com cert

---

https://github.com/h5bp/server-configs-nginx/pull/204 makes h5bp to reuse the expected host's ssl cert which is essentially the first part of point 2 (ssl.no-default.conf uses the first existing self-signed cert)

---

Can we make ssl.no-default to use one of the let's encrypt ssl cert or the manual ones?

Possible.

Should we do it?

Not until we solve:

1. the edge case: All sites using third party ssl providers
2. if the Trellis server has 2 https sites, which ssl cert to use

What are the possible solutions?

1. force the all third party ssl providers to be cognizant of ssl.no-default.conf
2. trellis tries to find a ssl cert (Let'sEncrypt, then manual, then self-signed) for ssl.no-default; if none found, create one; i.e: make #888 point 2 more intelligent

None of these solutions are good. Anyone has any suggestions?

[swalkinshaw]

I guess this goes back to the original issue; what's the problem with this extra certificate? It's been around forever now without any side effects or implications.

It's working and necessary (at least for now). Getting rid of it might require even more complexity.

I do have one thing I want to try/verify with Nginx regarding their default_server option.

[swalkinshaw]

Oh and @raph-topo, re:

So, we first have to solve #1073.

Not necessarily. We're free to structure our confs however we want and don't need to upgrade. We could easily just generate a standalone conf file with certicates in it only instead of them within the huge wordpress template.

[raph-topo]

Okay, I get your points.
I have no particular argument why this certificate must not be there. I was initially just surprised, as I do not remember ever seeing a second invalid certificate for example.com.
SSLabs reports A+ for recent Trellis, so I guess it is really no issue at all.

Nonetheless, because example.com exists, I replaced it with example.invalid on my servers, which is an actually invalid TLD.

• now there's a straight-forward thread about this for future readers wondering about this as well ;-)

[swalkinshaw]

I thought maybe using default_server in the real server blocks might work too but it doesn't.

Here's the only simpler solution I could find: https://stackoverflow.com/a/25280214. if is usually bad in Nginx but that's within location blocks. Problem is we'd need to conditionally check all hosts and Nginx doesn't support "OR" conditions. So we could generate a series of conditionals like this: https://gist.github.com/jrom/1760790

and just return 444 if none match. To be honest... it's not bad and we'd get rid of all this extra no-default conf complexity.

The other solution is to just extract the current ssl_certificate logic into its own conf and re-use it.

[swalkinshaw]

if ($host !~ "(^host1.test$)|(^host2.test$)") {
  return 444
}

We could just generate this easily 🤷‍♂️

edit: the wildcard subdomains make it harder 😞

[swalkinshaw]

Here's an example of this working (from my testing): https://github.com/roots/trellis/compare/simplify-nginx-no-default-host?expand=1

What do we think of this?

[raph-topo]

if (…) { return … } is "100% safe", says https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/:

The only 100% safe things which may be done inside if in a location context are:

• return ...;
• rewrite ... last;

From what I understand from https://serverfault.com/a/525011 and https://github.com/h5bp/server-configs-nginx/blob/c5c6602232e0976d9e69d69874aa84d2a2698265/sites-available/ssl.no-default, the server { … default_server … return 444 } should remain to prevent hijacks on enabled sites (as Nginx selects the first server block the configuration if there is no default_server.

My current understanding is that there are so many potential edge cases with Trellis, that #888 made the edge case the norm and cut short to dealing with edge cases by using an invalid certificate.

[raph-topo]

https://stackoverflow.com/a/43128186 also confirms this.

Can I propose the following:

• leave everything as is, since (according to the different comments):
  
  
  
  • it ignores all potential edge cases which would require complex configuration (which is error-prone thanks to Ansible);
  
  
  • it avoids hijacks on enabled host names;
  
  
  • does not expose any actual certificate for unknown host names;
  
  
  • the invalid certificate never shows up in modern browsing (i.e. always host name based);
  
  
• change the invalid certificate CN to your.request.is.invalid which makes it self-explanatory and clearly intentional (example.com looks like somebody forgot to configure something) — conforms with RFC2606:

".invalid" is intended for use in online construction of domain names that are sure to be invalid and which it is obvious at a glance are invalid.

• document this in the docs, so that intrigued users find the explanation for this.

[swalkinshaw]

Sure do you want to do the domain change? That's a simple thing we can now/separately.

I still think it's sort of useful to extract our certificates into a conf file and just re-use it in the no-default site instead of generating the self-signed one? But that can be a future thing if anyone wants to do that.

[raph-topo]

Sure do you want to do the domain change? That's a simple thing we can now/separately.

Sure, will do.

• [x] Domain name change.
• [ ] Docs about this.

I still think it's sort of useful to extract our certificates into a conf file and just re-use it in the no-default site instead of generating the self-signed one? But that can be a future thing if anyone wants to do that.

Sounds sensible. Could there be any implications in using the same certificate in many site? Maybe actually advantages?

[swalkinshaw]

Could there be any implications in using the same certificate in many site? Maybe actually advantages?

I think the only advantage is we skip generating one more cert 🤷‍♂️ which is probably very fast and not a big deal. If we're already doing the work for the no-default sites then generating the self-signed cert isn't a big deal. So maybe there's nothing more to do 😄

[raph-topo]

That's your call! 😄

The no-default certificate is only ever generated the first time the server is set up: https://github.com/roots/trellis/blob/2686136c9c635fa56991647ffd6635933438f523/roles/wordpress-setup/tasks/self-signed-certificate.yml#L25 introduced by https://github.com/roots/trellis/commit/15d453c8a0c3f5eab37c62f9d812fc001f18d5a7#diff-ff830fc010a506c1a47b7aba05cd8bb7R9 when introducing Let'sEncrypt.

Ansible:

creates | path | A filename or (since 2.0) glob pattern. If it already exists, this step won't be run.

This means, self-signed certificates do not get regenerated when their configuration changes?

[swalkinshaw]

This means, self-signed certificates do not get regenerated when their configuration changes?

Yeah correct.

[swalkinshaw]

Going to consider this closed after #1191. Led to a small improvement but interesting discussion 😄