Flynn: Automatic HTTPS through Let's Encrypt

[Detry322]

(This is an enhancement)

It would be really cool if it were possible to automatically enable https through the new Let's Encrypt service, since it's now trusted by all major browsers. It's also part of the Linux Foundation so it's rooted in the Open Source community.

[titanous]

Yeah, we'd love to have this. Ideally they'd have support for wildcard certificates, but that's not a blocker.

[philiplb]

This would be totally awesome. No more worries about expired certificates, no more manual requesting them.
They won't support wildcard afaik and the runtime is 90 days. They recommend to renew them actually like once per week? See https://letsencrypt.org/2015/11/09/why-90-days.html

[titanous]

Yeah, we'll have automated renewal set up.

[rikur]

+1 would save time and money when running multiple flynn environments (development, staging, production).

[temujin9]

:+1: - this would save me time, money, and frustration.

[christiangenco]

+1

[hadifarnoud]

+1 saves a lot of time.

[SystemZ]

Any news on this?
Any tip how to renew cert without downtime before auto renew is implemented?

[titanous]

@SystemZ this is currently scheduled for work as soon as user authentication (currently in progress) lands.

There should be no issues renewing Let's Encrypt certificates without downtime, issue a new certificate (I like the this tool) and then use flynn route update to do a zero-downtime replacement of the certificate.

[SystemZ]

@titanous Thanks for fast info and tip with acme :)

[slava-vishnyakov]

After some digging here's a workaround using Caddy until Flynn has LetsEncrypt built-in:

BEWARE! This is a dirty hack... Use at your own risk.

Seriously! Dirty hack with some hardcoding ahead!

On your local computer (where flynn CLI is):

export DB=$(flynn -a controller env get PGDATABASE)

flynn -a postgres pg psql -- $DB <<EOF
SELECT processes->'app'->'args' 
  FROM releases 
  WHERE processes->'app'->'args'->>0 = '/bin/flynn-router';
EOF

This should show you something like this:

["/bin/flynn-router", "-http-port", "80", "-https-port", "443", "-tcp-range-start", "3000", "-tcp-range-end", "3500"]

If your output is different - modify the next command to match.

Replacing 80 with 82 and 443 with 445 - issue this command:

flynn -a postgres pg psql -- $DB <<EOF
UPDATE releases 
  SET processes = jsonb_set(
    processes, '{app,args}', 
    jsonb '["/bin/flynn-router", "-http-port", "82", "-https-port", "445", "-tcp-range-start", "3000", "-tcp-range-end", "3500"]') 
  WHERE processes->'app'->'args'->>0 = '/bin/flynn-router';
EOF

This will move flynn-router from ports 80 to 82, and from 443 to 445, so that Caddy can run (otherwise it can't request certificates).

Then run (again on _local computer_):

echo "
cat <<EOF | sudo tee /etc/ssl/private/flynn.crt
$(flynn -a postgres pg psql -- -qAt $DB -c "select env->>'TLSCERT' from releases where processes->'app'->'args'->>0 = '/bin/flynn-router';")
EOF
cat <<EOF | sudo tee /etc/ssl/private/flynn.key
$(flynn -a postgres pg psql -- -qAt $DB -c "select env->>'TLSKEY' from releases where processes->'app'->'args'->>0 = '/bin/flynn-router';")
EOF
chmod 0600 /etc/ssl/private/flynn.*
"

This will output for you a set of instructions - you need to run _them_ on _flynn-host_ computer (i.e. where cluster is) -- this will export your "controller" certificates for Caddy (otherwise your flynn CLI will stop working, complaining about certificate pinning).

_Warning_ I have no idea when those expire and how they are managed, if you lose access to your controller - change your .flynnrc URLs (2 of them) to https://...:445 (That is also an alternative to exporting certificates, but the dashboard won't work, since it relies on port 443 heavily. And Caddy can't request ceritificates for *.flynnhub.com for some reason..)

After you run the above commands, run those on _flynn-host_ system:

wget -qO- https://getcaddy.com | bash

_Beware!_ bash|curl is lazy and insecure. It's better to install by hand! https://caddyserver.com/download

Then run those on _flynn-host_ replacing your e-mail with real one (for Let's Encrypt)

export [email protected]

cat <<EOT > /etc/init/caddy.conf
#!upstart

description "caddy"

start on runlevel [2345]
stop on runlevel [!2345]

console output

env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

exec /usr/local/bin/caddy --conf /etc/caddy/Caddyfile --agree
respawn
EOT

cat <<EOT > /etc/caddy/Caddyfile
https://* {
  tls /etc/ssl/private/flynn.crt /etc/ssl/private/flynn.key
  proxy / https://127.0.0.1:445 {
    insecure_skip_verify
    header_upstream Connection {>Connection}
    header_upstream Upgrade {>Upgrade}
    header_upstream Host {host}
    header_upstream X-Real-IP {remote}
    header_upstream X-Forwarded-For {remote}
    header_upstream X-Forwarded-Proto {scheme}
  }
}

example.domain.com othersite.com {
  tls $MY_EMAIL
  proxy / http://127.0.0.1:82 {
    header_upstream Connection {>Connection}
    header_upstream Upgrade {>Upgrade}
    header_upstream Host {host}
    header_upstream X-Real-IP {remote}
    header_upstream X-Forwarded-For {remote}
    header_upstream X-Forwarded-Proto {scheme}
  }
}
EOT

stop flynn-host
start flynn-host
start caddy

(Note: in Caddy 0.9.0 with this configuration the dashboard doesn't quite work. you need to download the latest git and build it to make dashboard (and websockets) work)

In latest (git master) Caddy this is enough (in 0.9.0 this doesn't work):

https://* {
  tls /etc/ssl/private/flynn.crt /etc/ssl/private/flynn.key
  proxy / http://127.0.0.1:82 {
    websocket
    transparent
  }
}

_Warning_ This will run Caddy as root, which is not recommended. Alternatively you can wget -qO /etc/init/caddy.conf https://raw.githubusercontent.com/mholt/caddy/master/dist/init/linux-upstart/caddy.conf, but this runs Caddy as www-data and that user will need to have access to your certificates. I don't know which would be better.

You will need to change example.domain.com (othersite.com is added just as example) to your hosted site in /etc/caddy/Caddyfile and restart caddy

(I'm not affiliated with Flynn team, so again - use at your own risk, this recipe might be incredibly stupid for all I know :) )

*_To revert: *_

on Flynn host:

stop caddy
rm /etc/init/caddy.conf

on local machine:

export DB=$(flynn -a controller env get PGDATABASE)
flynn -a postgres pg psql -- $DB <<EOF
UPDATE releases 
  SET processes = jsonb_set(
    processes, '{app,args}', 
    jsonb '["/bin/flynn-router", "-http-port", "80", "-https-port", "443", "-tcp-range-start", "3000", "-tcp-range-end", "3500"]') 
  WHERE processes->'app'->'args'->>0 = '/bin/flynn-router';
EOF

on Flynn host:

stop flynn-host
start flynn-host

[benosman]

I've seen the following golang library vendored a few times in other software to assist with LetsEncrypt / ACME integration. Supports HTTP and DNS validation.

https://github.com/xenolf/lego

Just putting it here in case you guys haven't come across it as yet.

[titanous]

@benosman Thanks! We're aware of the packages available and plan to use this one (I've sent a few patches already): https://godoc.org/golang.org/x/crypto/acme

[ghost]

Is this _dirty hack_ the only option to get let's encrypt support manually?

[titanous]

@dropfen no, you can use the -manual flag with DNS or HTTP (hosted on Flynn) from your local machine with this tool: https://github.com/google/acme

[ghost]

It sounds good, but I don't have a clue how to do it really :)

[ghost]

If someone knows what @titanous mean, please take some minutes if you can, to describe how to get letsencrypt work with flynn. Would be really helpful for a lot of people. thank you guys

[ghost]

@titanous Ok, it was really easy, actually, thank you for the tip 👍

[Banashek]

@dropfen care to post a small description? I'm just getting started with flynn (been using straight containers with compose) and would love to try out integrating letsencrypt.

[ghost]

Sure, no problem:

• follow the instructions here: https://github.com/google/acme (you should end up with cert & key in ~/.config/acme/)
• then update the route for the domain

flynn route <route-id> update --tls-key ~/.config/acme/your-domain.key --tls-cert ~/.config/acme/your-domain.crt

_hint: to get the route id, just call flynn route_

[mbreedlove]

@dropfen Won't this be a problem when you have to renew the certificate? It looks like you would have to manually stop the application to renew the certificate via Acme.

I've been using the Dokku Let's Encrypt plugin for a while, it just adds a new nginx location block to intercept the Let's Encrypt domain verification URL so the application can continue running, uninterrupted.

@titanous, is there a way to let the Acme client perform verification while the Flynn application is running?

[titanous]

@mbreedlove A few points:

1. The authorizations are currently valid for 10 months, so it's likely that the renewal will not require a new authorization and will be issued immediately (this has been my experience).
2. Path-based routing can be used to send the requests to a different app.
3. An endpoint can be added to the app for the LE HTTP authorization.
4. DNS authorization can be used.

None of these options require any downtime or interruption.

[ghost]

@mbreedlove to get a new cert you can use
acme cert -manual example.com
Then, you would have to put challenge file in the .well-known/acme-challenge dir and letsencrypt would just call it, to verify your host.
And finally with flynn route update you can put the new cert again into the router. So no downtime is required.

Note: you don't have to use a file, for me (phoenix framework) it didn't worked with .dot_folders like .well-known, but it worked with a route.. So, it would be also possible to keep your challenges in the DB and manage them over the UI, for example..

[philiplb]

What just came into my mind: Although I will totally use this, there might be people who won't. So this should be maybe opt-in? Also because it would let Flynn depend on a third party service which might go down the next day.

[titanous]

@philiplb It will be opt-out. The native security win is too compelling to require people to work to take advantage of it. I'm confident that they won't disappear tomorrow and if there are any issues we'll evaluate carefully.

[hadifarnoud]

is this going to be per route or per app? we're using one app with so many domains, per app would make it hard to use

[titanous]

@hadifarnoud Yes, per route.

[kations]

@titanous could you assume how long it takes to use this feature in prod?

[thomasmodeneis]

@dropfen the correct usage for adding the certificate has changed, now is (if you are in the same folder as your app is)
flynn route <route-id> update -k ~/.config/acme/your-domain.key -c ~/.config/acme/your-domain.crt

Or from any folder, using -a

flynn -a yourapp route <route-id> update -c ~/.config/acme/your-domain.crt -k ~/.config/acme/your-domain.key

[txchen]

This feature will be super useful for those who don't have a wildcard cert.

[michael-bouvy]

Very useful thread, thanks. Working command is:

flynn route update <route-id> -k <path-to-key>.pem -c <path-to-fullchain>.pem

[hadifarnoud]

@michael-bouvy how do I get letsencrypt to verify my domain? that's needed in order to get pem files

[michael-bouvy]

@hadifarnoud take a look at https://github.com/lukas2511/dehydrated

[michael-bouvy]

Indeed, I did use DNS challenge to get this working.

[hadifarnoud]

it works for some subdomains and don't for others even though I set all subdomains correctly. how can I reverse the route update?

[michael-bouvy]

flynn route remove <route-id>

[puterleat]

Is anyone actually working on LE integration right now? I'm weighing up the effort of hacking something together vs waiting and/or helping do it right. (EDIT - I cant see a LE branch, for example)

[nemanjan00]

Guys. This is the only reason I am using Dokku and not Flynn. :D

How can I help?

Please, can we speed this up?

Thank you!

[nemanjan00]

Dokku does this automatically, with one command.

Setting it all up is too slow.

It already takes my more than 2h to bootstrap each project I start...

[andyjeffries]

+1 We're using Dokku in production for our app and API (along with a number of personal projects) and I'd love not to be. The two reasons for that is that when I tried an older version of Flynn it had a tendency to grow disk space usage over time and never reclaim it and Lets Encrypt support. I'd love to help this feature that's been open for over a year (when both Caddy and Dokku support it out of the box).

[DoubleMalt]

Is there anyone working on it?

[clementvp]

Is there anyone working on it?

Dokku use a let's encrypt plug-in, and is so easy to get in, but doesn't support clustering.
Flynn is production ready and support clustering but doesn’t support let's encrypt.

It's a PaaS hard choice ;)

Wildcard certificate are SO expensive for little start up :/

[OmgImAlexis]

@clementvp let's encrypt is adding wildcard certs.

[clementvp]

@OmgImAlexis Not before january 2018 right ?

[girvo]

I'm also curious as to what the progress is like for this, or at least where I should start to look to add it myself! It's the only missing feature as far as what I require, so it'd be a massive boon to be able to get it in the core 😀

[titanous]

The current plan is to target ACME v2/wildcards, which will be landing in January. It's likely that implementation pieces will land before that.

[llim5432]

Any updates on this?

[roquie]

I also waiting for updates…

[shalakhin]

Me too

[lmars]

Work is in progress, I pushed a proof of concept last week (see this commit) which is functional but untested.

If you want to try it out in a test / staging environment then install version v20180105.dev1 and set LETS_ENCRYPT_KEY to a PEM encoded private EC key during bootstrap, for example:

export LETS_ENCRYPT_KEY="$(openssl ecparam -genkey -name prime256v1 -noout)"
export CLUSTER_DOMAIN="my-domain.com"
flynn-host bootstrap

[jvgreenaway]

@lmars how do you enable/test the https route? I have deployed the go-flynn-example app onto a v20180105.dev1 cluster and it is working with http but cannot work out how to enable https.

[lmars]

@jvgreenaway just try hitting the route with HTTPS, it should lazily go off and fetch the certificate. Do note though that it will probably no longer work for new domains due to Let's Encrypt disabling TLS-SNI.

[darkpixel]

You might look at this for a work-around: https://github.com/containous/traefik/issues/1832

It sounds like they'll be disabling it permanently. Some projects have implemented the DNS method via plugins for services like Digital Ocean, Route 53, etc...others have enabled HTTP verification, etc...

[jvgreenaway]

Thank you for your ideas.

Hmm – I've had a play but there doesn't seem to be a turnkey option at the moment. It seems I would need to turn off the flynn service and bring down all my apps to validate certs for new apps. Would this approach have consequences when using on a cluster?

[seospace]

Can anyone tell what is the current-fastest way of doing this?

[lmars]

I have updated my proof of concept to support http-01 challenges, released as version v20180216.dev3.

You can try it out as follows:

• install and start version v20180216.dev3

curl -fsSLo install-flynn https://dl.flynn.io/install-flynn
sudo bash install-flynn --version "v20180216.dev3"
sudo systemctl start flynn-host

• bootstrap with LETS_ENCRYPT_KEY set

export LETS_ENCRYPT_KEY="$(openssl ecparam -genkey -name prime256v1 -noout)"
export CLUSTER_DOMAIN="example.com"
flynn-host bootstrap

• deploy an app, for example nodejs-flynn-example

git clone https://github.com/flynn-examples/nodejs-flynn-example.git
cd nodejs-flynn-example
flynn create hello
git push flynn master

• hit the default app route with HTTPS, after a short while you should see it gets a cert

curl -v https://hello.example.com
...
* Server certificate:
*  subject: CN=hello.example.com
*  start date: Feb 16 23:10:16 2018 GMT
*  expire date: May 17 23:10:16 2018 GMT
*  subjectAltName: host "hello.example.com" matched cert's "hello.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.

A more user-friendly version is being worked on.

[nemanjan00]

Looks like I will be switching to flynn soon :D

[NishidaRyu416]

@lmars
Can I also renew automatically?

[christiaangoossens]

Any indication on when this will go into stable with auto issuing and renewal?

[bmv437]

Good news, ACME v2 with Wildcard support was officially launched yesterday!

https://community.letsencrypt.org/t/acme-v2-and-wildcard-certificate-support-is-live/55579

[roquie]

How long to wait release that look at this feature in nightly builds? We really want this release! :)

[david-szabo97]

Any update? Is this implemented? I still can't decide between Dokku and Flynn, but the lack of Let's Encrypt support is a bit holding me back. I only need it for a small side project so I think I will just go with Dokku in this case. But I would like to see this get implemented, mostly because of cluster support. :)

[txchen]

@david-szabo97 check out https://github.com/githubsaturn/captainduckduck
Maybe this can fit your requirements.

[david-szabo97]

@txchen Guess what, I was just looking at this project in this very moment 😄

[ariefrizkyr]

Any updates regarding this issue? It is very nice to see the integration between Let's Encrypt and Flynn

[francocorreasosa]

Hey guys! Any updates on this? Thank you

[4ware]

I would love to use flynn on an upcomming project but the lack of certificate generation is a showstopper.

Any news on this?

[nrempel]

Hey everyone, I needed this as well so I wrote a little script that can be deployed on your Flynn cluster to automatically manage this.

I thought I would share this since it seems like a popular requirement.

https://github.com/nrempel/flynn-certbot

It runs certbot to obtain certs and runs flynn-cli to update routes. It's configured with environment variables. Currently it only works with Digital Ocean but other certbot plugins should be fairly easy to add support for.

[francocorreasosa]

You definitely got my star on that. Thank you 🎉

[philiplb]

@nrempel added to https://github.com/philiplb/awesome-flynn :)

[roquie]

52 days left before the four-anniversary date of this issue. :disappointed:

Any news about certificates?

[guestisp]

Updates?

[matteing]

Is this alive?

A fully functioning Certbot integration for Flynn would be a huge boon to the service overall.

@lmars @titanous

[iosifnicolae2]

I've tried to solve the problem in https://github.com/iosifnicolae2/flynn-certbot-standlone .
The application is not ready for production usage!
Any commit is appreciated.

[ec-raphael]

This is basically the main thing that's making me hesitant to use Flynn at the moment.

[sicutdeux]

Bump on this!, i see that the fix is being forwared to release .dev, any of you guys can answer if this works; https://github.com/flynn/flynn/releases/tag/v20191106.dev2 the problem i see references for using ACME tls-sni which is deprecated from feb-2019 so this method will not work, i have a project to deploy this week and it will be good if i can use flynn but the issue with lets is blocking me.

[matteing]

Flynn is pretty much dead.

On Nov 12, 2019 12:45 PM, sicutdeux [e-mail] wrote:

Bump on this!, i see that the fix is being forwared to release .dev, any of you guys can answer if this works; https://github.com/flynn/flynn/releases/tag/v20191106.dev2 the problem i see references for using ACME tls-sni which is deprecated from feb-2019 so this method will not work, i have a project to deploy this week and it will be good if i can use flynn but the issue with lets is blocking me.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/flynn/flynn/issues/1995?email_source=notifications&email_token=AB4XGZIRCTO4OHH6TSILNA3QTLMULA5CNFSM4BSJHKU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED24WEQ#issuecomment-552979218, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB4XGZJEAPUJ2SIZKFT72MDQTLMULANCNFSM4BSJHKUQ.

[sicutdeux]

You think? i was looking for something like flynn or dokku for clustering, can you recommend something?

[matteing]

Sadly. The parent company doesn’t seem to maintain it anymore in favor of enterprise customers. It’s sad because Flynn really is one-of-a-kind software.

On Nov 12, 2019, at 1:10 PM, sicutdeux <[email protected][e-mail]> wrote:

You think? i was looking for something like flynn or dokku for clustering, can you recommend something?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/flynn/flynn/issues/1995?email_source=notifications&email_token=AB4XGZIQQAEBW4DZINXLPZTQTLPQHA5CNFSM4BSJHKU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED27SRQ#issuecomment-552991046, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB4XGZLZDVYYYFCZ3UK6PJ3QTLPQHANCNFSM4BSJHKUQ.

[iosifnicolae2]

You think? i was looking for something like flynn or dokku for clustering, can you recommend something?

I built something on top of Dokku here: https://infrastructure.mailo.dev

[fr3fou]

any news?

[darkpixel]

If there was news, I'm sure it would be mentioned in the issue.

[clementvp]

Yeah just go to Caprover a long time ago :/

[fr3fou]

Yeah just go to Caprover a long time ago :/

it's what i've been using for a year now, it's good

[scratchmex]

I can't believe such a great project is dead.

[hadifarnoud]

it's outdated tech now. give K3S a try. so simple and elegant

[titanous]

Flynn is unmaintained and our infrastructure will shut down on June 1, 2021. See the README for details.

[darkpixel]

Sorry to see you guys go. The software was great.