Lego: Improvement: setting dns-01 challenge record for multiple domains in parallel

[JorritSalverda]

When running lego for multiple domains it will set the dns TXT record and check propagation in sequence. This can take a long time if the number of domains is large. Would it be possible to parallelize this?

[xenolf]

Well it sure is possible. @mholt and me already had a talk about this and we came to agree that it would be a great feature. The ACME challenges are good to run concurrent as the context for each of them should be seperate. What we need to do though is look all over to be sure that no state is shared.

So yes, this is something I'd like to add in the future :)

[lenovouser]

Yeah, this would be very helpful if you have a lot of domains.

[mthoretton]

Just another comment to add you some additional pressure about developing this great functionality ;) Btw lego works fine with route53! Thanks

[xenolf]

I've started to work on this (for all challenges) locally and will push once I have something to show ;)

[xi2]

Thank you for working on this. I will be happy to test and/or update the gandi provider as required. :)

[aaronb-cg]

I would like to see this also.
Lego is working great for us, just takes a long time with the large number of domains we're processing.

[jasonwbarnett]

Would love to collaborate to get this done. I looked over the code tonight and identified where the modification should go, but haven't had time to brainstorm the best design, e.g. worker pool, go routines, etc... Thinking that because the Let's Encrypt limit is 100 domains it's probably safe to spawn 100 goroutines lazily. thoughts?

Function that needs updating: https://github.com/xenolf/lego/blob/master/acme/client.go#L510-L523

Note: I could imagine there are circumstances where some forms of ACME challenges (e.g. HTTP (http-01)) need to be run serially because you can only bind to a single port one at a time. Don't actually know if this is true because I don't know how lego implements the other challenge types, but I wanted to call it out so we (I 😄 ) don't blindly make this concurrently for all challenge types.

[jasonwbarnett]

I leveraged #237 as a starting point and was able to make things even more efficient. I was able to obtain a SSL certificate with 86 SANs in 6 minutes and 24 seconds.

I was using the following:

• Challenge Type: DNS-01
• DNS Provider: Google DNS
• TXT Record TTL: 10 seconds
• Zone negative cache TTL: 10 seconds

Will clean up and publish within a day or so

[xenolf]

Hey @jasonwbarnett and thanks for your time!
I had a quick look at your implementation. Please keep in mind that many of the DNS providers currently shipped with lego do have state per instance of the provider (for caching the domain name for example). Your implementation only calls the solve() function in parallel which will overwrite this state.

[jasonwbarnett]

@xenolf Where did you get with your implementation? Do you have it published anywhere?

Wondering if it makes sense to implement the concurrency in the provider itself or what... Or if it's just a matter of refactoring the providers which currently have a shared state.

[xenolf]

@jasonwbarnett My approach was to create one instance of the provider for each call to solve(). The code isn't published anywhere but I can push it into a branch once I get home.

[jasonwbarnett]

@xenolf Please post what you have to a branch so I can see.

[lenovouser]

Any progress?

[jasonwbarnett]

@lenovouser I did it for google on my fork here and it works perfectly, https://github.com/jasonwbarnett/lego/tree/feature/concurrent-solvers

[lenovouser]

@jasonwbarnett can you create a pull request? Or did you already?

[aslafy-z]

up? (I know it's bad, but...)

[jasonwbarnett]

I'm up :)

@aslafy-z what are you asking?

[aslafy-z]

Will you PR this awesome feature ?

[jasonwbarnett]

Since I last worked on this I am 105 commits behind master and last I looked (6 months ago) the changes were fairly drastic and this would take a bit of work to integrate into the latest.

I will have some time later this week at which point I'll spend some time looking into rebasing off of master and getting this thing working. I doubt it will get merged in by @xenolf though...

[xenolf]

@jasonwbarnett What makes you think that?

[jasonwbarnett]

:innocent: I stand corrected

[ldez]

Fixed by #607