Certbot: Document the new authentication hooks in the manual plugin

[pde]

• Markdown docs from #3521 should be ported to RST and added to docs/using.rst
• Add some of the --hooks-* flags to the "script" CLI help topic
• Add the plugin to the table of plugins in docs/using.rst

[pde]

@pconrad-fb interested in taking this?

[pde]

Closing this ticket for now; will reopen once #3767 refactoring is [maybe] done.

[pconrad-fb]

Yes, this looks doable once the refactoring is done.

[pde]

OK, #3890 has landed, so we now have the task of providing good user-facing docs on how to use the --manual-auth-hook and --manual-cleanup-hook flags.

[pde]

This is an 0.10.0 ticket but it's probably okay if this happens shortly after the release.

[bmw]

@SwartzCr, we discussed possibly giving you this ticket if you're interested on the Certbot call this morning. Would you like to work on it or should I?

[SwartzCr]

@bmw if you can give me a rundown I'm happy to work on it

[bmw]

Awesome. There's actually a decent amount of documentation already written, but it's spread around throughout our repo and needs to be cleaned up. Take a look at the comments in #3521 as well as the output from certbot --help manual from master.

For the most part, everything is as Joona describes in #3521. The only real change is the script plugin was merged into the manual plugin. References to --script should be changed to --manual, --script-auth to --manual-auth-hook, and --script-cleanup to --manual-cleanup-hook.

Basically, the way the manual plugin works now is if the user provides a shell command or a path to an executable with --manual-auth-hook, the manual plugin behaves like the script plugin Joona describes. If this flag is not provided, the manual plugin works like it always did; simply giving the user instructions on how to set up their system to pass the challenge from the ACME CA themselves.

We should explain these two use cases in the style we do for our other plugins.

I think there's a lot of nice things we could do for explaining the use case with hooks, however, some might be a bit tricky and I don't think they are strictly necessary. Ideally, for the use case with hooks, we should include an example that other people can use to as a starting point and to build their own scripts off of. While more complicated, I think doing a DNS example here makes the most sense as this how I expect most people would be using this script. Taking it a step further, it'd be nice if this example was for a real API like Cloudflare, Namecheap, etc. This gives people a real example to work with and allows users of those services to simply copy and paste the script we provide in our documentation.

[pconrad-fb]

I'd love to help out as well. I'm out of the office right now, returning next week, and I have some cycles to spare.

[pconrad-fb]

So... should I sync and start work? I don't want to step on anyone else's changes and create complex merge scenarios, but if no one is taking this I can start Thursday.

[bmw]

@pconrad-fb, I think we have a couple options for things for you to work on:

1. Noah just took a first pass at this issue in #3975. If you want to try and polish what he wrote that would be valuable as this is a new feature of Certbot that's a little complicated to use. The easier we can make this to understand the better!
2. In a similar vein, @ohemorange recently added a number of new subcommands to Certbot. The new subcommands are certificates, delete, and update_symlinks. At least the first two (but perhaps update_symlinks as well) should be documented at https://certbot.eff.org/docs/using.html. If you have a working version of certbot, you can learn more about these commands through certbot --help. If you don't have a working version of certbot or you have additional questions, her or I can give you a summary of what these commands do so you can write more user facing documentation.

Let me know what you'd like to do!

EDIT: Erica also added a --cert-name flag which is an important thing to document for our users. More information on this stuff (not all of which is new) can be found in the output of certbot --help manage.

[pconrad-fb]

Perfect. I'll sync my fork and start by doing a polish pass.

[pconrad-fb]

I get a merge conflict on install.rst. Looks like there's been some work done there, possibly some of what I put in was deprecated out. Should I bother to resolve it, or should I just pull the file and wipe out what I have locally?

EDIT:
Never mind, I will resolve and then do the polish. I looked more closely at the conflict and I see what's going on.

[pconrad-fb]

Interesting. I installed certbot (brew install certbot) and it seemed to go okay, but something must be up with my Python because certbot --help gives me a stack trace (pasted below in case you want it). I can try it on my Linux box... but perhaps if you can send me the output of certbot --help it'll move things along quicker.

EDIT: I installed on a fairly fresh Ubuntu 16.10 upgrade (after running apt-get update) and I don't see the new subcommands. Can you send whatever information you have on them? FYI, I'm running 0.8.1-1 and a repeat install says that's the latest version.

pconrad-ltm6:installations pconrad$ certbot --help
Traceback (most recent call last):
File "/usr/local/bin/certbot", line 11, in
load_entry_point('certbot==0.10.0', 'console_scripts', 'certbot')()
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/pkg_resources/__init__.py", line 561, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
return ep.load()
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2291, in load
return self.resolve()
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2297, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/certbot/main.py", line 13, in
from acme import jose
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/acme/jose/__init__.py", line 37, in
from acme.jose.interfaces import JSONDeSerializable
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/acme/jose/interfaces.py", line 9, in
from acme.jose import util
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/acme/jose/util.py", line 5, in
import OpenSSL
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in
from OpenSSL import rand, crypto, SSL
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in
from OpenSSL._util import (
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in
from cryptography.hazmat.bindings.openssl.binding import Binding
File "/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in
from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: dlopen(/usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so, 2): Symbol not found: _getentropy
Referenced from: /usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so
Expected in: /usr/lib/libSystem.B.dylib
in /usr/local/Cellar/certbot/0.10.0/libexec/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

[bmw]

Sorry for all the trouble :frowning_face:

brew install certbot should normally work, however, it looks there is a problem which someone else also reported in #4067. Hopefully this is resolved soon.

The version of Certbot in Ubuntu's repos unfortunately isn't up to date and doesn't include the new subcommands. You can use certbot-auto to get a new enough version though by following our instructions for older versions of Ubuntu.

@ohemorange expressed interest to me about potentially writing up a brain dump on how the new subcommands work. I'll see if she's still interested in doing that, but here's Certbot's relevant help output:

manage:
  Various subcommands and flags are available for managing your
  certificates:

  certificates          List certificates managed by Certbot
  delete                Clean up all files related to a certificate
  update_symlinks       Recreate symlinks in your /etc/letsencrypt/live/
                        directory
  --cert-name CERTNAME  Certificate name to apply. Only one certificate name
                        can be used per Certbot run. To see certificate names,
                        run 'certbot certificates'. When creating a new
                        certificate, specifies the new certificate's name.
                        (default: None)

[pconrad-fb]

Cool, a brain dump would be great source material. Meanwhile, I'll see what I can do in terms of inserting information into the docs. What release number are these commands available in?

[bmw]

The commands are new in Certbot 0.10.0.

[ohemorange]

@pconrad-fb @bmw Drafted docs into using.rst at https://github.com/certbot/certbot/pull/4092, would love review/revision/reworking as appropriate.

[bmw]

@pconrad-fb, if you're still interested, I'd love for you to take a look at this. You can either review it and suggest changes yourself or I can review it and you can do a follow up PR after it's merged. Whichever you prefer.

[pconrad-fb]

Let me pull the changes down and take a look.

[pconrad-fb]

Oh, right, it won't make it into my fork unless it's accepted into the repo. I'm looking online in the git diff view, but I'm not git-savvy enough to know how to pull this version into my local copy. I'll see if I can figure it out.

[pconrad-fb]

Other than a minor typo on line 545, my main feedback is that this reads like a list of commands and what they do, rather than a list of what to do and how to do it (with the commands). The naive user coming in will want to know that they can obtain, revoke, renew, etc. and will then want to understand the subcommands and flags to express (and refine) that goal. With that in mind, I'd make "Renewing Certificates" a section underneath the new "Managing Certificates" (and move the resulting "Managing" up to where "Renewing" was) and add sections about revoking, etc.

I'm happy to tackle this work, but I'm not sure how to get the new content in a way that won't result in a merge problem, short of it being pushed to master where I can grab it for my fork.

[ohemorange]

(@SwartzCr)

@pconrad-fb try:

git checkout master
git pull
git checkout managing-docs
git checkout -b pconrad-managing-docs

After making changes:

git add -A
git commit -m "Updates to the docs"
git push -u origin pconrad-managing-docs

Then create a pull request from the github website.

Those suggestions sound great! Would you mind taking a pass at them?

[pconrad-fb]

Will do.

UPDATE:
Still will do, a couple work things came up, but I'm trying to hit my target for volunteer hours by the end of January, so in order to do that I have to get this next pass done this week. So, probably I'll get some work done on it tomorrow or Thursday.

[pconrad-fb]

@ohemorange I've made changes to the file, git commit -am "What I did" doesn't complain, and git status is clean. Now I'm not sure how to either (a) merge to my own master, or (b) create the managing-docs branch on my fork—or, more importantly, which you prefer for the pull request. Can you give me some guidance?

[ohemorange]

@pconrad-fb Do (b), please! Try: git push -u origin managing-docs.

[pconrad-fb]

Okay, did that... now when I look to create a pull request, I get:

There isn’t anything to compare.
certbot:managing-docs and pconrad-fb:managing-docs are identical.

So... my local copies look like they have the latest changes, my git commit has nothing to commit, my git status is clean, so I wonder if you have my changes already as well. Look for "0.10.0" in using .rst and if you find it, it's got my changes in it. Otherwise, uh, let me know how to get git to give them to you.

[ohemorange]

What does git log say?

[pconrad-fb]

commit bf7ce3130298016bdeb970e7c8e04cf319099969
Author: Peter Conrad [e-mail]
Date: Thu Jan 26 15:53:48 2017 -0800

Restructuring and editing using.rst to incorporate new command docs better.

commit a1702e766d4ecda952e614cd482b79fb5bc912d4
Author: Erica Portnoy [e-mail]
Date: Fri Jan 20 11:35:40 2017 -0800

Add information about cert management to the docs

commit 02615c2ac67570330ec8faf2ded9421c4efa4bc6
Author: Nick Fong [e-mail]
Date: Fri Jan 20 09:40:36 2017 -0800

Silence Package Manager Output when certbot-auto invoked with --quiet (#3776)

* Add quiet flags to package manager invocations

Add the following flags when 'certbot-auto --quiet' is invoked:
- Add '-qq' to calls to 'apt-get' in Debian
- Add '--quiet' to calls to 'yum' or 'dnf' in CentOS or Fedora
- Add '--quiet' to calls to 'urpmi' in Mageia
- Add '--quiet' to calls to 'pkg install' in FreeBSD

* Fix $QUIET flag in bootstrappers

- Set the value of $QUIET properly (i.e. s/$QUIET/QUIET when setting the
  variable) in
  - deb_common.sh
  - mageia_common.sh
  - rpm_common.sh
- Actually use $QUIET when running $tool in rpm_common.sh

* Add handling of $QUIET to Arch and Open Suse

[pconrad-fb]

What's the best way for me to get these changes to you?

[ohemorange]

Looks like it didn't manage to push properly. Could you hop on IRC or any other chat application? Hangouts?

[pconrad-fb]

I am in a meeting, I can type in hangouts but it would be hard to actually videoconference. You can ping me at [email protected]

[SwartzCr]

@pconrad-fb did you get your git issues worked out? If not feel free to ping me and I can help work them out!

[ohemorange]

@SwartzCr, we worked that out! #4137

[SwartzCr]

@ohemorange great - I take it it'd help if I reviewed #4137?

[ohemorange]

that would be great actually! I probably won't be able to get to it until Tuesday unfortunately.

[pconrad-fb]

I'd love to get a review. Happy to make changes as needed, or move on to the next set. I'll watch this thread.

[SwartzCr]

I made comments on #4137, which I expect @ohemorange to review - your copy editing was great, just a few larger clarification issues.
@pconrad-fb if you'd like something to work on, I'd suggest taking a look at: #4153
If that one doesn't appeal to you, there's also #4152 and $4016, but I thought #4153 (which deals with our DNS challenge) would be a good one since it has to do with the auth-hooks that were in the last documentation PR.
As always feel free to ping me if you have any questions or need guidance

[pde]

https://certbot.eff.org/docs/using.html#hooks is live; closing!