Kong: Error uploading certificate on 0.14.0 : schema violation, required field missing

[pixeye33]

Summary

Can't upload a certificate on a brand new install of kong (docker alpine)

Steps To Reproduce

1. try to add a certificate on a brand new install of kong (docker alpine)

here are the files i'm trying to upload
fullchain.cer (DER BASE64 formated) :
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

private.key (DER BASE64 formated) :
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----

both files are obtained via acme.sh and letsencrypt.

Additional Details & Logs

• Kong version : 0.14.0
• Kong configuration (pretty much default out of the box) :
  {"plugins":{"enabled_in_cluster":[],"available_on_server":{"response-transformer":true,"oauth2":true,"acl":true,"correlation-id":true,"pre-function":true,"jwt":true,"cors":true,"ip-restriction":true,"basic-auth":true,"key-auth":true,"rate-limiting":true,"request-transformer":true,"http-log":true,"file-log":true,"hmac-auth":true,"ldap-auth":true,"datadog":true,"tcp-log":true,"zipkin":true,"post-function":true,"request-size-limiting":true,"bot-detection":true,"syslog":true,"loggly":true,"azure-functions":true,"udp-log":true,"response-ratelimiting":true,"aws-lambda":true,"statsd":true,"prometheus":true,"request-termination":true}},"tagline":"Welcome to kong","configuration":{"plugins":["bundled"],"admin_ssl_enabled":false,"lua_ssl_verify_depth":1,"trusted_ips":{},"prefix":"/usr/local/kong","loaded_plugins":{"response-transformer":true,"request-termination":true,"prometheus":true,"ip-restriction":true,"pre-function":true,"jwt":true,"cors":true,"statsd":true,"basic-auth":true,"key-auth":true,"ldap-auth":true,"aws-lambda":true,"http-log":true,"response-ratelimiting":true,"hmac-auth":true,"request-size-limiting":true,"datadog":true,"tcp-log":true,"zipkin":true,"post-function":true,"bot-detection":true,"acl":true,"loggly":true,"syslog":true,"azure-functions":true,"udp-log":true,"file-log":true,"request-transformer":true,"correlation-id":true,"rate-limiting":true,"oauth2":true},"cassandra_username":"kong","admin_ssl_cert_csr_default":"/usr/local/kong/ssl/admin-kong-default.csr","ssl_cert_key":"/usr/local/kong/ssl/kong-default.key","dns_resolver":{},"pg_user":"kong","mem_cache_size":"128m","cassandra_data_centers":["dc1:2","dc2:3"],"nginx_admin_directives":{},"custom_plugins":{},"pg_host":"db-kong","nginx_acc_logs":"/usr/local/kong/logs/access.log","proxy_listen":["0.0.0.0:8000","0.0.0.0:8443 ssl"],"client_ssl_cert_default":"/usr/local/kong/ssl/kong-default.crt","ssl_cert_key_default":"/usr/local/kong/ssl/kong-default.key","dns_no_sync":false,"db_update_propagation":0,"nginx_err_logs":"/usr/local/kong/logs/error.log","cassandra_port":9042,"dns_order":["LAST","SRV","A","CNAME"],"dns_error_ttl":1,"headers":["server_tokens","latency_tokens"],"dns_stale_ttl":4,"nginx_optimizations":true,"database":"postgres","pg_database":"kong","nginx_worker_processes":"auto","lua_package_cpath":"","admin_acc_logs":"/usr/local/kong/logs/admin_access.log","lua_package_path":"./?.lua;./?/init.lua;","nginx_pid":"/usr/local/kong/pids/nginx.pid","upstream_keepalive":60,"admin_access_log":"/dev/stdout","client_ssl_cert_csr_default":"/usr/local/kong/ssl/kong-default.csr","proxy_listeners":[{"ssl":false,"ip":"0.0.0.0","proxy_protocol":false,"port":8000,"http2":false,"listener":"0.0.0.0:8000"},{"ssl":true,"ip":"0.0.0.0","proxy_protocol":false,"port":8443,"http2":false,"listener":"0.0.0.0:8443 ssl"}],"proxy_ssl_enabled":true,"cassandra_contact_points":["127.0.0.1"],"enabled_headers":{"latency_tokens":true,"X-Kong-Proxy-Latency":true,"Via":true,"server_tokens":true,"Server":true,"X-Kong-Upstream-Latency":true,"X-Kong-Upstream-Status":false},"ssl_ciphers":"ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256","cassandra_ssl":false,"db_resurrect_ttl":30,"ssl_cert_csr_default":"/usr/local/kong/ssl/kong-default.csr","cassandra_consistency":"ONE","client_max_body_size":"0","admin_error_log":"/dev/stderr","pg_ssl_verify":false,"dns_not_found_ttl":30,"pg_ssl":false,"cassandra_schema_consensus_timeout":10000,"client_ssl":false,"cassandra_repl_strategy":"SimpleStrategy","db_update_frequency":5,"nginx_kong_conf":"/usr/local/kong/nginx-kong.conf","cassandra_repl_factor":1,"nginx_http_directives":[{"value":"prometheus_metrics 5m","name":"lua_shared_dict"}],"kong_env":"/usr/local/kong/.kong_env","real_ip_header":"X-Real-IP","dns_hostsfile":"/etc/hosts","log_level":"notice","error_default_type":"text/plain","ssl_cert":"/usr/local/kong/ssl/kong-default.crt","cassandra_lb_policy":"RoundRobin","admin_ssl_cert_key_default":"/usr/local/kong/ssl/admin-kong-default.key","cassandra_ssl_verify":false,"proxy_access_log":"/dev/stdout","ssl_cipher_suite":"modern","real_ip_recursive":"off","proxy_error_log":"/dev/stderr","client_ssl_cert_key_default":"/usr/local/kong/ssl/kong-default.key","nginx_daemon":"off","anonymous_reports":true,"nginx_proxy_directives":{},"cassandra_timeout":5000,"pg_port":5432,"admin_listeners":[{"ssl":false,"ip":"0.0.0.0","proxy_protocol":false,"port":8001,"http2":false,"listener":"0.0.0.0:8001"}],"client_body_buffer_size":"8k","lua_socket_pool_size":30,"admin_ssl_cert_default":"/usr/local/kong/ssl/admin-kong-default.crt","db_cache_ttl":0,"cassandra_keyspace":"kong","ssl_cert_default":"/usr/local/kong/ssl/kong-default.crt","nginx_conf":"/usr/local/kong/nginx.conf","admin_listen":["0.0.0.0:8001"]},"version":"0.14.0","node_id":"486e327b-bdaa-4cc6-940b-637025e3911c","lua_version":"LuaJIT 2.1.0-beta3","prng_seeds":{"pid: 49":103181771581,"pid: 45":235178194225,"pid: 46":107142381512,"pid: 47":188163461912,"pid: 48":729392136818,"pid: 42":893511320512,"pid: 43":617712621371,"pid: 44":150424450101},"timers":{"pending":5,"running":0},"hostname":"a368ca5d8bb6"}
• Operating system : Docker container

on acme.sh side :
{"fields":{"cert":"required field missing","key":"required field missing"},"name":"schema violation","code":2,"message":"2 schema violations (cert: required field missing; key: required field missing)"}

on my kong container side :
app-kong_1 | 2018/07/11 17:57:37 [notice] 43#0: *800586 [lua] arguments.lua:656: load(): request body already exists, client: 172.21.0.5, server: kong_admin, request: "POST /certificates HTTP/1.1", host: "app-kong:8001"
app-kong_1 | 172.21.0.5 - - [11/Jul/2018:17:57:37 +0000] "POST /certificates HTTP/1.1" 400 228 "-" "acme.sh/2.7.9 (https://github.com/Neilpang/acme.sh)"

[stevenkitter]

the some question,who can response.
curl -i -X POST http://localhost:8001/certificates
-F "cert=@/path/to/cert.pem"
-F "key=@/path/to/cert.key"
-F "snis=ssl-example.com,other-ssl-example.com"
got {"fields":{"cert":"required field missing","key":"required field missing"},"name":"schema violation","code":2,"message":"2 schema violations (cert: required field missing; key: required field missing)"}

[Donald42]

I'm having the same issue. I have a script that worked perfectly fine on my slightly older Kong installation, but the one I set up today will not install certs at all.

[pixeye33]

Hello, can someone actually respond ? @bungle ? @thibaultcha ?

If you need more info, please let us know.
If we're noobs doing it wrong, make fun of us (while telling us why).
I'm willing to (try to) fix it, anyone has pointers on where the issue can be ?

[jeremyjpj0916]

Decided to take 5 mins and test it on 0.14.0 even though I don't use cert resources, works fine just trying strings..

/ $ curl -i -XPOST localhost:8001/certificates --data 'key=----BEGIN CERTIFICATE----' --data 'cert=----BEGIN RSA PRIVATE KEY----'
HTTP/1.1 201 Created
Date: Fri, 20 Jul 2018 08:01:37 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Access-Control-Allow-Origin: *
Content-Length: 153

{"created_at":1532073697,"cert":"----BEGIN RSA PRIVATE KEY----","id":"55c3f2bd-68f0-47ab-bcc1-b26531d0f0ee","key":"----BEGIN CERTIFICATE----","snis":[]}
/ $

Exhaust all methods of POSTS before giving up :P !

Also -F "snis=ssl-example.com,other-ssl-example.com" , your not passing a file there, so why would that work? you will need to pass properly as a list.

-d snis[]=hurrr-d snis[]=derrrr

Hope that helps.

[pixeye33]

well, thanks for your answer !!
indeed it does work...

and :
konga doesn't work (fixed in v0.12.0)
acme.sh doesn't work (https://github.com/Neilpang/acme.sh/blob/master/deploy/kong.sh)

current doc is not really specific : https://docs.konghq.com/0.14.x/admin-api/#add-certificate

maybe there was a change on kong side beyond just the path used for the api : https://docs.konghq.com/plugins/dynamic-ssl/
old way was :
curl -X POST http://kong:8001/services/{service}/plugins
--data "name=ssl"
--data "config.cert=@/path/to/cert.pem"
--data "config.key=@/path/to/cert.key"
--data "config.only_https=true"

[bungle]

@pixeye33, I think it had to do with multipart/form-data.

[bungle]

The issue here is that Lapis (which is used for admin api) consumes the request body, and the multipart cannot be anymore stream parsed (as what our admin api arguments parse would do, so this was a known issue). We didn't need multipart support until now. The certificates where moved to new dao and that's why we are having this issue now. We need to fix the multipart parsing (a short term fix could be just using lua-multipart there (that doesn't stream parse).

[stevenkitter]

/ $ curl -i -XPOST localhost:8001/certificates --data 'key=----BEGIN CERTIFICATE----' --data 'cert=----BEGIN RSA PRIVATE KEY----'
that is ok. 'key=----BEGIN CERTIFICATE----' do not use terminal.I recommand postman,just paste the file's content.

[JeremyShort]

When is this going to be fixed or the example updated? Can't use Kong with out TLS support.

[bungle]

@JeremyShort as said, you can upload your certificate using application/x-www-form-urlencoded instead of multipart/form-data. We are working on fixing this.

[ionosphere80]

Using curl with -H "Content-Type: application/json" and JSON content appears to work.

[bungle]

This is now fixed, well 80% fixed (we will revisit it later), as we merged this:

3776

I will close it for now. Please re-open if you find that this was not properly solved.