Trellis: letsencrypt in unfixable state if first run is done before pointing DNS to server

[vesper8]

Submit a feature request or bug report

• [x ] I've read the guidelines for Contributing to Roots Projects
• [ ] This is a feature request
• [X ] This is a bug report
• [ ] This request isn't a duplicate of an existing issue
• [ x] I've read the docs and followed them (if applicable)
• [ x] This is not a personal support request that should be posted on the Roots Discourse forums

Replace any X with your information.

---

What is the current behavior?

I accidentally ran the provision script on a new domain with SSL enabled before I actually changed the DNS to point to the server. This failed of course, but it also created an unfixable loop of errors in regards to the letsencrypt task.

What is the expected or desired behavior?

After pointing the dns to the server and running the provision script again, the playbook should have been able to repair itself

---

Bug report

I accidentally ran the provision script on a new domain with SSL enabled before I actually changed the DNS to point to the server. This failed of course, but it also created an unfixable loop of errors in regards to the letsencrypt task.

Afterwards I pointed the DNS to the server for this failed site and no matter what I did (including waiting over an hour) it would always fail with the Acme Test

If I removed the ssl flag, then it would complete correctly and the site would be available with http://

Re-adding the SSL flag would result in the same errors

failed: [myhost] (item=mydomain.com) => {"changed": false, "failed_hosts": ["mydomain.com"], "item": "mydomain.com", "rc": 1}

Finally what fixed it was SSH'ing to the server and deleting this file:

/etc/nginx/sites-available/letsencrypt-mydomain.com.conf

That file had been created only once, the first time I ran the playbook, when the DNS wasn't pointing to the server anymore.

Now the crazy thing is.. I don't understand why deleting this file made the problem go away. Also, this file was NOT re-created when it finally fixed the issue.. which worries me

The site is now accessible via https however and the playbook isn't returning errors anymore

Would love to know what is happening here and why this was happening

[fullyint]

Hi @vesper8! That would be a really frustrating situation. Glad you figured it out.

I suspect this was an unfortunate glitch you'll never encounter again, but if you can share steps to help me reliably reproduce the problem, I'd love to get it resolved.

If that letsencrypt-mydomain.com.conf was indeed the cause, I suspect this was a rare glitch because
1) I don't recall ever seeing this issue before
2) I know that I and others often run the letsencrypt role before DNS has been set up or propagated, requiring nothing more than rerunning server.yml with --tags letsencrypt once DNS is ready
3) a few details about the letsencrypt-mydomain.com.conf file...
* If ever the conf is enabled, Trellis notifies a task that should always run later to disable the conf (a "handler"). This handler's cleanup would only fail if the playbook were manually interrupted (e.g., CTRL+c) or if the playbook were to experience some other abnormal failure that prevents handlers from running.
* The wordpress-setup role _always_ imports and runs that same task to disable temporary challenge sites. You ran the wordpress-setup role when you disabled SSL temporarily, so I don't know why it wouldn't have disabled the conf for you.

One potential source of the problem: If after first failed playbook run, the site key was changed such that subsequent cleanup attempts wouldn't look for nor find the filename to clean up (filename includes site key).

---

I don't understand why deleting this file made the problem go away. Also, this file was NOT re-created when it finally fixed the issue

If the letsencrypt-mydomain.com.conf file had the same server_name value as your primary /etc/nginx/sites-available/mydomain.com.conf then Nginx would only pick one or the other server {} blocks, probably leading to something unexpected. When you removed the conf, you perhaps removed the competing server block (if somehow that sites-available/letsencrypt-mydomain.com.conf you removed was still symlinked over to sites-enabled/letsencrypt-mydomain.com.conf).

The letsencrypt-mydomain.com.conf file's purpose is just to include acme-challenge-location.conf; if /etc/nginx/sites-available/mydomain.com.conf doesn't already exist with that include directive, which is typically only the case on your very first run of server.yml. Subsequent runs usually don't need this include directive added, so the letsencrypt-mydomain.com.conf file is usually not needed nor created.

---

I'm going to close this because I doubt it's very reproducible, but feel free to post back with more info or share steps to help me reliably reproduce the problem and we'll reopen.

[vesper8]

kudos to you @fullyint for the nicest, most detailed and sympathetic explanation of what might have gone wrong and why! seriously you should win an award for writing like this!

I don't know what happened.. it might have been a coincidence in the end.. I was trying to figure out if letsencrypt has its own dns cache or something.. because the ping and dig commands could all confirm that the dns had already propagated but the letsencrypt kept failing

perhaps it would be a nice addition to add a ping check prior to initiating the letsencrypt for a domain.. so that if the dns isn't yet pointing to the server, it skips it entirely?