Fabio: Consul K/V certstore and wildcard certificates

[kaspergrubbe]

Hi,

I am a little bit confused about how the SSL-certs works with Consul, I have the following domains

billett.ooo

and

*.test.mothership.billett.ooo

If I run Fabio like this:

/usr/bin/fabio \
-proxy.cs 'cs=ssl;type=consul;cert=http://localhost:8500/v1/kv/ssl-certificates'

And then configure my services with the following tags:

tags = ["urlprefix-fabio.test.mothership.billett.ooo/ proto=https"]

and:

tags = ["urlprefix-billett.ooo/ proto=https", "urlprefix-www.billett.ooo/ proto=https"]

Should the pem/key be placed at ssl-certificates/billett.ooo.pem and ssl-certificates/billett.ooo.key? And should I also have a cert like ssl-certificates/www.billett.ooo.pem even though it is the same cert for both www and non-www?

And how would I install and configure the wildcard certificate?

[leprechau]

Fabio takes certificates in a combined PEM format. With consul the path you specify is interpreted as a prefix. In your example it's expected that you would have a structure similar to this in your Consul KV store:

/ssl-certificates/
                 /cert1
                 /cert2

In each of those keys you should have a concatenated combined pem ... if cert1 was a file you would create it by doing:

cat domain-cert.pem intermediate-signing-cert.pem domain-cert.key > cert1

That make sense? With LetsEncrypt it would be the fullchain bundle concatenated with the private key.

[kaspergrubbe]

@leprechau Hi Aaron, thanks for getting back to me.

That does make sense, does Fabio read the domains from the certificates and use the correct one automatically based on the request, or do I need to name my keys accordingly?

[leprechau]

The KV and/or file name isn't important. Fabio will pick the most specific matching CN of the available certificates.

[kaspergrubbe]

@leprechau That is good to know, that sounds almost magical :)

I am having some issues and the log-file isn't helping me so much.

My service is tagged with: "urlprefix-clusterapp.prod.mothership.billett.ooo/ proto=https".

I've installed the fullchain certificate inside of consul and it lives at kv/sslcerts/cert01:

And I am using the default Fabio configuration, only thing changed is the proxy.cs like this:

proxy.cs = cs=billettocerts;type=consul;cert=http://localhost:8500/v1/kv/sslcerts

However, the log doesn't show anything related to the proxy.cs setting, and when visiting the site https://clusterapp.prod.mothership.billett.ooo/ I get an error:

Here is the full fabio log:

2019/03/15 14:20:31 [INFO] Setting log level to INFO
2019/03/15 14:20:31 [INFO] Runtime config
{
    "Proxy": {
        "Strategy": "rnd",
        "Matcher": "prefix",
        "NoRouteStatus": 404,
        "MaxConn": 10000,
        "ShutdownWait": 0,
        "DialTimeout": 30000000000,
        "ResponseHeaderTimeout": 0,
        "KeepAliveTimeout": 0,
        "FlushInterval": 1000000000,
        "GlobalFlushInterval": 0,
        "LocalIP": "10.1.1.192",
        "ClientIPHeader": "",
        "TLSHeader": "",
        "TLSHeaderValue": "",
        "GZIPContentTypes": null,
        "RequestID": "",
        "STSHeader": {
            "MaxAge": 0,
            "Subdomains": false,
            "Preload": false
        }
    },
    "Registry": {
        "Backend": "consul",
        "Static": {
            "NoRouteHTML": "",
            "Routes": ""
        },
        "File": {
            "NoRouteHTMLPath": "",
            "RoutesPath": ""
        },
        "Consul": {
            "Addr": "localhost:8500",
            "Scheme": "http",
            "Token": "",
            "KVPath": "/fabio/config",
            "NoRouteHTMLPath": "/fabio/noroute.html",
            "TagPrefix": "urlprefix-",
            "Register": true,
            "ServiceAddr": ":9998",
            "ServiceName": "fabio",
            "ServiceTags": null,
            "ServiceStatus": [
                "passing"
            ],
            "CheckInterval": 1000000000,
            "CheckTimeout": 3000000000,
            "CheckScheme": "http",
            "CheckTLSSkipVerify": false,
            "CheckDeregisterCriticalServiceAfter": "90m",
            "ChecksRequired": "one"
        },
        "Timeout": 10000000000,
        "Retry": 500000000
    },
    "Listen": [
        {
            "Addr": ":9999",
            "Proto": "http",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "CertSource": {
                "Name": "",
                "Type": "",
                "CertPath": "",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "Header": null
            },
            "StrictMatch": false,
            "TLSMinVersion": 0,
            "TLSMaxVersion": 0,
            "TLSCiphers": null
        }
    ],
    "Log": {
        "AccessFormat": "common",
        "AccessTarget": "",
        "RoutesFormat": "delta",
        "Level": "INFO"
    },
    "Metrics": {
        "Target": "",
        "Prefix": "{{clean .Hostname}}.{{clean .Exec}}",
        "Names": "{{clean .Service}}.{{clean .Host}}.{{clean .Path}}.{{clean .TargetURL.Host}}",
        "Interval": 30000000000,
        "Timeout": 10000000000,
        "Retry": 500000000,
        "GraphiteAddr": "",
        "StatsDAddr": "",
        "Circonus": {
            "APIKey": "",
            "APIApp": "fabio",
            "APIURL": "",
            "CheckID": "",
            "BrokerID": ""
        }
    },
    "UI": {
        "Listen": {
            "Addr": ":9998",
            "Proto": "http",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "CertSource": {
                "Name": "",
                "Type": "",
                "CertPath": "",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "Header": null
            },
            "StrictMatch": false,
            "TLSMinVersion": 0,
            "TLSMaxVersion": 0,
            "TLSCiphers": null
        },
        "Color": "light-green",
        "Title": "",
        "Access": "rw"
    },
    "Runtime": {
        "GOGC": 800,
        "GOMAXPROCS": 1
    },
    "ProfileMode": "",
    "ProfilePath": "/tmp",
    "Insecure": false,
    "GlobMatchingDisabled": false
}
2019/03/15 14:20:31 [INFO] Version 1.5.10 starting
2019/03/15 14:20:31 [INFO] Go runtime is go1.11.1
2019/03/15 14:20:31 [INFO] Running fabio as UID=0 EUID=0 GID=0
2019/03/15 14:20:31 [WARN] 

  ************************************************************
  You are running fabio as root without the '-insecure' flag
  This will stop working with fabio 1.7!
  ************************************************************

2019/03/15 14:20:31 [INFO] Metrics disabled
2019/03/15 14:20:31 [INFO] Setting GOGC=800
2019/03/15 14:20:31 [INFO] Setting GOMAXPROCS=1
2019/03/15 14:20:31 [INFO] Running fabio as UID=0 EUID=0 GID=0
2019/03/15 14:20:31 [WARN] 

  ************************************************************
  You are running fabio as root without the '-insecure' flag
  This will stop working with fabio 1.7!
  ************************************************************

2019/03/15 14:20:31 [INFO] consul: Connecting to "localhost:8500" in datacenter "eu-central-1"
2019/03/15 14:20:31 [INFO] Admin server access mode "rw"
2019/03/15 14:20:31 [INFO] Admin server listening on ":9998"
2019/03/15 14:20:31 [INFO] Waiting for first routing table
2019/03/15 14:20:31 [INFO] consul: Using dynamic routes
2019/03/15 14:20:31 [INFO] consul: Using tag prefix "urlprefix-"
2019/03/15 14:20:31 [INFO] consul: Watching KV path "/fabio/config"
2019/03/15 14:20:31 [INFO] consul: Watching KV path "/fabio/noroute.html"
2019/03/15 14:20:31 [INFO] HTTP proxy listening on :9999
2019/03/15 14:20:31 [INFO] Running fabio as UID=0 EUID=0 GID=0
2019/03/15 14:20:31 [WARN] 

  ************************************************************
  You are running fabio as root without the '-insecure' flag
  This will stop working with fabio 1.7!
  ************************************************************

2019/03/15 14:20:31 [INFO] Access logging disabled
2019/03/15 14:20:31 [INFO] Using routing strategy "rnd"
2019/03/15 14:20:31 [INFO] Using route matching "prefix"
2019/03/15 14:20:31 [INFO] consul: Registered fabio as "fabio"
2019/03/15 14:20:31 [INFO] consul: Registered fabio with id "fabio-ip-10-1-1-192.eu-central-1.compute.internal-9998"
2019/03/15 14:20:31 [INFO] consul: Registered fabio with address "10.1.1.192"
2019/03/15 14:20:31 [INFO] consul: Registered fabio with tags ""
2019/03/15 14:20:31 [INFO] consul: Registered fabio with health check to "http://[10.1.1.192]:9998/health"
2019/03/15 14:20:32 [INFO] Config updates
+ route add whoami www.billett.ooo/ http://10.1.2.147:25262/
+ route add whoami www.billett.ooo/ http://10.1.1.192:31507/
+ route add whoami www.billett.ooo/ http://10.1.1.192:29010/
+ route add whoami billett.ooo/ http://10.1.2.147:25262/
+ route add whoami billett.ooo/ http://10.1.1.192:31507/
+ route add whoami billett.ooo/ http://10.1.1.192:29010/
+ route add nomad-ui nomad.prod.mothership.billett.ooo/ http://10.1.3.173:4646/ opts "allow=ip:192.168.0.42/32 pxyproto=true"
+ route add nomad-ui nomad.prod.mothership.billett.ooo/ http://10.1.2.178:4646/ opts "allow=ip:192.168.0.42/32 pxyproto=true"
+ route add nomad-ui nomad.prod.mothership.billett.ooo/ http://10.1.1.243:4646/ opts "allow=ip:192.168.0.42/32 pxyproto=true"
+ route add consul-ui consul.prod.mothership.billett.ooo/ http://10.1.3.5:8585/ opts "allow=ip:192.168.0.42/32 pxyproto=true"
+ route add consul-ui consul.prod.mothership.billett.ooo/ http://10.1.2.107:8585/ opts "allow=ip:192.168.0.42/32 pxyproto=true"
+ route add consul-ui consul.prod.mothership.billett.ooo/ http://10.1.1.156:8585/ opts "allow=ip:192.168.0.42/32 pxyproto=true"
+ route add clusterapp-group-web clusterapp.prod.mothership.billett.ooo/ https://10.1.2.147:22822
2019/03/15 14:20:33 [INFO] Config updates
+ fabio-ui fabio.prod.mothership.billett.ooo/ http://10.1.2.147:9998/ opts "allow=ip:192.168.0.42/32"
+ route add
2019/03/15 14:20:35 [INFO] Config updates
+ fabio-ui fabio.prod.mothership.billett.ooo/ http://10.1.1.192:9998/ opts "allow=ip:192.168.0.42/32"
+ route add
invalid log msg: 2019/03/15 14:21:48 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:21:52 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:21:54 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:22:02 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:22:02 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:22:06 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:22:08 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:22:12 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:22:12 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:23:02 http: proxy error: tls: oversized record received with length 20527
invalid log msg: 2019/03/15 14:23:03 http: proxy error: tls: oversized record received with length 20527
invalid log msg: 2019/03/15 14:23:04 http: proxy error: tls: oversized record received with length 20527
invalid log msg: 2019/03/15 14:23:48 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:26:09 http: proxy error: context canceled
invalid log msg: 2019/03/15 14:26:09 http: proxy error: context canceled

Am I missing something obvious?

[leprechau]

You'll need the fullchain + key in the key store (in the cert01 key). The server needs the certificate (public key) chain plus the private key to initiate an encrypted connection.

[shantanugadgil]

Hi, just a hunch but wildcard certs I have seen are usually *.domain.tld and not for *.*.*.domain.tld

Would it possible to verify the cert and key first using a tls web server like
https://github.com/udhos/gowebhello

[kaspergrubbe]

@shantanugadgil Thanks for your suggestion. I did have some issues with the wildcard certs, so I've decided to just use full domains and skip the wildcard certs for now, it is more important to get some https working than get wildcards working :) I will take a look at gowebhello when I toy with wildcard certs again.

Edit: These are the domains in my certificate:

[kaspergrubbe]

@leprechau I see, I didn't read what you wrote, I have now done the cat fullchain.pem privkey.pem > cert01.pem and pasted it into Consul, but still no change.

I noticed this from the logs:

route add clusterapp-group-web clusterapp.prod.mothership.billett.ooo/ https://10.1.2.147:22822

Does that look right? It look like Fabio is doing a https connection to my service, I thought Fabio would serve the cert and do http internally.

[leprechau]

Ohh, okay ... so fabio can do HTTPs termination for you on the front ... and it can also do HTTPs termination to backend services. The proto=https option to the urlprefix- tag tells fabio it should use https when communicating with the backend.

I think what you are wanting is for fabio to handle all HTTPs termination and then use plain HTTP to the backend? If that's the case you simply need to configure the urlprefix- without the proto=https option and connect to fabio via an HTTPs listener. To do that you need to configure the certificate store like you have done AND specify an HTTPs listener via the proxy.addr option:

https://fabiolb.net/ref/proxy.addr/

You can also have fabio automatically redirect HTTP to HTTPS using a redirect.

https://fabiolb.net/feature/http-redirects/

Hope that helps!

[leprechau]

Here's an example configuration from our development environment where we have several listeners defined:

proxy.cs = cs=files;type=path;cert=/etc/fabio/certs;refresh=60s
proxy.addr = 11.22.33.65:80;proto=http;rt=90s;wt=60s,\
             11.22.33.65:443;proto=https;rt=90s;wt=60s;cs=files;tlsmin=tls10, \
             11.22.33.68:80;proto=http;rt=69s;wt=60s,\
             11.22.33.68:443;proto=https;rt=90s;wt=60s;cs=files;tlsmin=tls10, \
             11.22.33.68:8443;proto=tcp+sni, \
             11.22.33.65:26257;proto=tcp

The above config creates one certificate store as a file base where I have the concatenated certs in /etc/fabio/certs. It also defines multiple listeners some service HTTPS on port 443, HTTP on port 80, TCP+SNI passthrough on port 8443 and standard TCP on port 26257.

Requests inbound on port 80 will be standard HTTP ... requests on port 443 fabio will terminate TLS and us the configured certificate store to find a certificate matching the SNI of the request.

To force HTTPS we place the following in the consul KV config:

route add http-redirect dev.ena.net:80 https://dev.ena.net$path opts "redirect=301"

[kaspergrubbe]

@leprechau Thank you so much for your patience and examples, very handy to know about the \ for newlines, and the http redirection.

I've now done the changes you recommended, so my config is now like this:

proxy.cs = cs=billettocerts;type=consul;cert=http://127.0.0.1:8500/v1/kv/sslcerts
proxy.addr = :9999;proto=http,:9995;proto=https;cs=billettocerts

And now Fabio also picks up the configuration on boot:

{
    "Addr": ":9995",
    "Proto": "https",
    "ReadTimeout": 0,
    "WriteTimeout": 0,
    "CertSource": {
        "Name": "billettocerts",
        "Type": "consul",
        "CertPath": "http://127.0.0.1:8500/v1/kv/sslcerts",
        "KeyPath": "",
        "ClientCAPath": "",
        "CAUpgradeCN": "",
        "Refresh": 0,
        "Header": null
    },
    "StrictMatch": false,
    "TLSMinVersion": 0,
    "TLSMaxVersion": 0,
    "TLSCiphers": null
}

However, when I visit any of the sites the log shows this:

2019/03/16 06:24:20 [INFO] cert: Store has certificates for [""]
invalid log msg: 2019/03/16 06:33:38 http: TLS handshake error from 139.162.191.30:61064: cert: no certificates stored

It turned out that the key had to be named cert01.pem instead of just cert01 to work 👍