Kong: Can't get Kong 0.14.0 to serve my certificates on HTTPS

[itaimalek]

Summary

After configuring Certs with SNIs Kong route is ignoring the configuration and serves localhost cert (default)

Steps To Reproduce

1. Configure .pem and .key strings to Certificate resource in Kong with SNI
2. Configure a Service and Route objects with Host in Route as in above SNI with https enabled for the Route
3. Try to access via https

Expected result

Kong will serve the certificate that it's SNI fits the Host of the Route.

Additional Details & Logs

• Kong version - 0.14.0
  setup is on kubernetes with postgres DB.
  HTTP access is good- I get response as expected.

my Service object:

{
            "host": "some.service.domain",
            "connect_timeout": 60000,
            "id": <service-id>,
            "protocol": "http",
            "name": <service-name>,
            "read_timeout": 60000,
            "port": 80,
            "path": "/",
            "updated_at": 1534887459,
            "retries": 5,
            "write_timeout": 60000
        }

my Route object:

{
    "created_at": 1534938341,
    "strip_path": false,
    "hosts": [
        "host.some.domain"
    ],
    "preserve_host": false,
    "regex_priority": 0,
    "updated_at": 1534938341,
    "paths": [
        "/some-path"
    ],
    "service": {
        "id": <service-id>
    },
    "methods": [
        "GET",
        "POST"
    ],
    "protocols": [
        "http",
        "https"
    ],
    "id": <route ID>
}

my SNI object:

{
            "certificate": {
                "id": <cert object ID>
            },
            "created_at": 1534852534,
            "name": "host.some.domain",
            "id": <SNI ID>
        }

[ionosphere80]

How did you add the certificate(s) to Kong?

[itaimalek]

Hi,
Thanks for replying,
I added certificates just like any other object, via kong admin api.

[ionosphere80]

Right... but did you use a JSON file with the application/json content type?

[itaimalek]

yes, I did a POST request with body like:

{ "cert": "<.pem file content>", "key": "<key file content>", "snis": [ "<host.some.domain>" ] }
also, the SNI was created from this request.
I didn't use a json file, the JSON object was sent via the request itself.
Also when doing GET to kong-admin-api/certificates, I can see my certificate there.

[itaimalek]

adding my kong configurations:

{
    "plugins": {
        "enabled_in_cluster": [],
        "available_on_server": {
            "response-transformer": true,
            "oauth2": true,
            "acl": true,
            "correlation-id": true,
            "pre-function": true,
            "jwt": true,
            "cors": true,
            "ip-restriction": true,
            "basic-auth": true,
            "key-auth": true,
            "rate-limiting": true,
            "request-transformer": true,
            "http-log": true,
            "file-log": true,
            "hmac-auth": true,
            "ldap-auth": true,
            "datadog": true,
            "tcp-log": true,
            "zipkin": true,
            "post-function": true,
            "request-size-limiting": true,
            "bot-detection": true,
            "syslog": true,
            "loggly": true,
            "azure-functions": true,
            "udp-log": true,
            "response-ratelimiting": true,
            "aws-lambda": true,
            "statsd": true,
            "prometheus": true,
            "request-termination": true
        }
    },
    "tagline": "Welcome to kong",
    "configuration": {
        "plugins": [
            "bundled"
        ],
        "admin_ssl_enabled": true,
        "lua_ssl_verify_depth": 1,
        "trusted_ips": {},
        "prefix": "/usr/local/kong",
        "loaded_plugins": {
            "response-transformer": true,
            "request-termination": true,
            "prometheus": true,
            "ip-restriction": true,
            "pre-function": true,
            "jwt": true,
            "cors": true,
            "statsd": true,
            "basic-auth": true,
            "key-auth": true,
            "ldap-auth": true,
            "aws-lambda": true,
            "http-log": true,
            "response-ratelimiting": true,
            "hmac-auth": true,
            "request-size-limiting": true,
            "datadog": true,
            "tcp-log": true,
            "zipkin": true,
            "post-function": true,
            "bot-detection": true,
            "acl": true,
            "loggly": true,
            "syslog": true,
            "azure-functions": true,
            "udp-log": true,
            "file-log": true,
            "request-transformer": true,
            "correlation-id": true,
            "rate-limiting": true,
            "oauth2": true
        },
        "cassandra_username": "kong",
        "admin_ssl_cert_csr_default": "/usr/local/kong/ssl/admin-kong-default.csr",
        "ssl_cert_key": "/usr/local/kong/ssl/kong-default.key",
        "admin_ssl_cert_key": "/usr/local/kong/ssl/admin-kong-default.key",
        "dns_resolver": {},
        "pg_user": "kong",
        "mem_cache_size": "128m",
        "cassandra_data_centers": [
            "dc1:2",
            "dc2:3"
        ],
        "nginx_admin_directives": {},
        "custom_plugins": {},
        "pg_host": "postgres",
        "nginx_acc_logs": "/usr/local/kong/logs/access.log",
        "proxy_listen": [
            "0.0.0.0:8000",
            "0.0.0.0:8443 ssl"
        ],
        "client_ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
        "ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
        "dns_no_sync": false,
        "db_update_propagation": 0,
        "nginx_err_logs": "/usr/local/kong/logs/error.log",
        "cassandra_port": 9042,
        "dns_order": [
            "LAST",
            "SRV",
            "A",
            "CNAME"
        ],
        "dns_error_ttl": 1,
        "headers": [
            "server_tokens",
            "latency_tokens"
        ],
        "dns_stale_ttl": 4,
        "nginx_optimizations": true,
        "database": "postgres",
        "pg_database": "kong",
        "nginx_worker_processes": "auto",
        "lua_package_cpath": "",
        "admin_acc_logs": "/usr/local/kong/logs/admin_access.log",
        "lua_package_path": "./?.lua;./?/init.lua;",
        "nginx_pid": "/usr/local/kong/pids/nginx.pid",
        "upstream_keepalive": 60,
        "cassandra_contact_points": [
            "127.0.0.1"
        ],
        "admin_access_log": "/dev/stdout",
        "client_ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr",
        "proxy_listeners": [
            {
                "ssl": false,
                "ip": "0.0.0.0",
                "proxy_protocol": false,
                "port": 8000,
                "http2": false,
                "listener": "0.0.0.0:8000"
            },
            {
                "ssl": true,
                "ip": "0.0.0.0",
                "proxy_protocol": false,
                "port": 8443,
                "http2": false,
                "listener": "0.0.0.0:8443 ssl"
            }
        ],
        "proxy_ssl_enabled": true,
        "pg_password": "******",
        "cassandra_ssl": false,
        "enabled_headers": {
            "latency_tokens": true,
            "X-Kong-Proxy-Latency": true,
            "Via": true,
            "server_tokens": true,
            "Server": true,
            "X-Kong-Upstream-Latency": true,
            "X-Kong-Upstream-Status": false
        },
        "ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr",
        "client_ssl": false,
        "db_resurrect_ttl": 30,
        "error_default_type": "text/plain",
        "cassandra_consistency": "ONE",
        "client_max_body_size": "0",
        "admin_error_log": "/dev/stderr",
        "pg_ssl_verify": false,
        "dns_not_found_ttl": 30,
        "pg_ssl": false,
        "db_update_frequency": 5,
        "ssl_ciphers": "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256",
        "cassandra_repl_strategy": "SimpleStrategy",
        "cassandra_repl_factor": 1,
        "log_level": "notice",
        "admin_ssl_cert": "/usr/local/kong/ssl/admin-kong-default.crt",
        "real_ip_header": "X-Real-IP",
        "kong_env": "/usr/local/kong/.kong_env",
        "cassandra_schema_consensus_timeout": 10000,
        "dns_hostsfile": "/etc/hosts",
        "admin_listeners": [
            {
                "ssl": false,
                "ip": "0.0.0.0",
                "proxy_protocol": false,
                "port": 8001,
                "http2": false,
                "listener": "0.0.0.0:8001"
            },
            {
                "ssl": true,
                "ip": "0.0.0.0",
                "proxy_protocol": false,
                "port": 8444,
                "http2": false,
                "listener": "0.0.0.0:8444 ssl"
            }
        ],
        "cassandra_timeout": 5000,
        "ssl_cert": "/usr/local/kong/ssl/kong-default.crt",
        "proxy_access_log": "/dev/stdout",
        "admin_ssl_cert_key_default": "/usr/local/kong/ssl/admin-kong-default.key",
        "cassandra_ssl_verify": false,
        "ssl_cipher_suite": "modern",
        "cassandra_lb_policy": "RoundRobin",
        "real_ip_recursive": "off",
        "proxy_error_log": "/dev/stderr",
        "client_ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
        "nginx_daemon": "off",
        "anonymous_reports": true,
        "db_cache_ttl": 0,
        "nginx_proxy_directives": {},
        "pg_port": 5432,
        "nginx_kong_conf": "/usr/local/kong/nginx-kong.conf",
        "client_body_buffer_size": "8k",
        "lua_socket_pool_size": 30,
        "admin_ssl_cert_default": "/usr/local/kong/ssl/admin-kong-default.crt",
        "nginx_http_directives": [
            {
                "value": "prometheus_metrics 5m",
                "name": "lua_shared_dict"
            }
        ],
        "cassandra_keyspace": "kong",
        "ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
        "nginx_conf": "/usr/local/kong/nginx.conf",
        "admin_listen": [
            "0.0.0.0:8001",
            "0.0.0.0:8444 ssl"
        ]
    },
    "version": "0.14.0",
    "node_id": "4d52dcd3-420c-4482-840f-110d9d3eb274",
    "lua_version": "LuaJIT 2.1.0-beta3",
    "prng_seeds": {
        "pid: 57": 921281731435,
        "pid: 58": 236194211245,
        "pid: 60": 188183197199,
        "pid: 59": 159121015489
    },
    "timers": {
        "pending": 5,
        "running": 0
    },
    "hostname": "kong-rc-6c4dbfbd9c-l8n4n"
}

[ionosphere80]

Did your certificate contain \n in place of CRs?

[itaimalek]

Did your certificate contain \n in place of CRs?
.
nope -verified that.

you can see Kong is serving a default certificate:

[ionosphere80]

If using JSON, the certificate and key objects should use \n in place of CRs. The Kong API has a tendency to accept certificates/keys with an incorrect format, but won't enable them.

[itaimalek]

NICE !
That did it !

Thanks allot !

[ionosphere80]

No problem. Kong should do a better job of validating body content.

[benjaminprojas]

I am having the same symptoms as the original issue here. My certificates are being loaded properly (I think), and yet kong isn't picking them up. When it returns the created or updated response, I see the line breaks are \n like @ionosphere80 mentioned.

I used to add these certificates in 0.13.x like this:

curl -i -X POST http://localhost:8001/certificates/ \
  -F 'cert=@/link/to/cert.pem' \
  -F 'key=@/link/to/key.pem' \
  -F 'snis=domain.com'

But if I do that now, it returns a "missing key" error. If I use JSON instead, it seems to load just fine, but it doesn't use it properly on the endpoint. Anything I am missing here?

[benjaminprojas]

I forgot to mention the way I am doing it now:

curl -i -X POST http://localhost:8001/certificates/ \
  -H 'Content-Type: application/json' \
  -d "{\"cert\": \"$(cat /link/to/cert.pem")\", \"key\": \"$(cat /link/to/key.pem")\", \"snis\": [ \"domain.com\" ] }"

[benjaminprojas]

Never mind, turns out I was generating the certs improperly. Thanks!

[DevOpsRoot]

Thanks @benjaminprojas, you save my day :100:

[bungle]

I feel like this is resolved now. Btw. the multipart/form-data support was added back with this (well 80% solution, but should work especially for this case):
https://github.com/Kong/kong/pull/3776

[xiaoshumiao6]

that is a so big trap