Rabbitmq-server: Improve TLS option validation
As requested in mailing list thread - opening an issue to improve error messages.
The error messages from rabbitmq are fairly gnarly. In my specific case the error message returned provided no discernable clue that I had a configuration error.
The log message:
Failed to start Ranch listener {acceptor,{0,0,0,0,0,0,0,0},5671} in ranch_ssl:listen([{port,
5671},
{ip,
{0,
0,
0,
0,
0,
0,
0,
0}},
inet6,
binary,
{packet,
raw},
{reuseaddr,
true},
{backlog,
128},
{nodelay,
true},
{exit_on_close,
false},
{versions,
['tlsv1.2',
'tlsv1.1',
tlsv1]},
{cacertfile,
"/etc/pki/tls/certs/organizationvalidation-sha-2-w-root.ca-bundle"},
{certfile,
"/etc/pki/tls/certs/rabbitmq.pem"},
{keyfile,
"/etc/pki/tls/private/rabbitmq.key"},
{verify,
true},
{fail_if_no_peer_cert,
false}]) for reason {options,
{verify,
true}} (unknown POSIX error)
=INFO REPORT==== 6-Apr-2016::15:47:33 ===
Error description:
{could_not_start,rabbit,
{{case_clause,
{error,
{{shutdown,
{failed_to_start_child,
{ranch_listener_sup,
{acceptor,{0,0,0,0,0,0,0,0},5671}},
{shutdown,
{failed_to_start_child,ranch_acceptors_sup,
{listen_error,
{acceptor,{0,0,0,0,0,0,0,0},5671},
{options,{verify,true}}}}}}},
{child,undefined,'rabbit_tcp_listener_sup_:::5671',
{tcp_listener_sup,start_link,
[{0,0,0,0,0,0,0,0},
5671,ranch_ssl,
[inet6,binary,
{packet,raw},
{reuseaddr,true},
{backlog,128},
{nodelay,true},
{exit_on_close,false},
{versions,['tlsv1.2','tlsv1.1',tlsv1]},
{cacertfile,
"/etc/pki/tls/certs/organizationvalidation-sha-2-w-root.ca-bundle"},
{certfile,"/etc/pki/tls/certs/rabbitmq.pem"},
{keyfile,"/etc/pki/tls/private/rabbitmq.key"},
{verify,true},
{fail_if_no_peer_cert,false}],
rabbit_connection_sup,[],
{rabbit_networking,tcp_listener_started,
['amqp/ssl']},
{rabbit_networking,tcp_listener_stopped,
['amqp/ssl']},
1,"SSL Listener"]},
transient,infinity,supervisor,
[tcp_listener_sup]}}}},
[{rabbit_networking,start_listener0,5,
[{file,"src/rabbit_networking.erl"},{line,318}]},
{rabbit_networking,'-start_listener/5-lc$^0/1-0-',5,
[{file,"src/rabbit_networking.erl"},{line,306}]},
{rabbit_networking,start_listener,5,
[{file,"src/rabbit_networking.erl"},{line,306}]},
{rabbit_networking,'-boot_ssl/1-lc$^0/1-0-',3,
[{file,"src/rabbit_networking.erl"},{line,147}]},
{rabbit_networking,boot_ssl,1,
[{file,"src/rabbit_networking.erl"},{line,147}]},
{rabbit_networking,boot,0,
[{file,"src/rabbit_networking.erl"},{line,133}]},
{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,
[{file,"src/rabbit_boot_steps.erl"},{line,49}]},
{rabbit_boot_steps,run_step,2,
[{file,"src/rabbit_boot_steps.erl"},{line,49}]}]}}
My rabbitmq.config ( changed the hostnames )
% This file managed by Puppet
% Template Path: rabbitmq/templates/rabbitmq.config
[
{rabbit, [
{cluster_nodes, {['[email protected]', '[email protected]'], disc}},
{cluster_partition_handling, ignore},
{tcp_listen_options,
[binary,
{packet, raw},
{reuseaddr, true},
{backlog, 128},
{nodelay, true},
{exit_on_close, false}]
},
{ssl_listeners, [5671]},
{ssl_options, [
{cacertfile,"/etc/pki/tls/certs/organizationvalidation-sha-2-w-root.ca-bundle"},
{certfile,"/etc/pki/tls/certs/rabbitmq.pem"},
{keyfile,"/etc/pki/tls/private/rabbitmq.key"},
{verify,true},
{fail_if_no_peer_cert,false}
]},
{default_user, <<"guest">>},
{default_pass, <<"guest">>}
]},
{kernel, [
]}
,
{rabbitmq_management, [
{listener, [
{port, 15671},
{ssl, true},
{ssl_opts, [
{cacertfile, "/etc/pki/tls/certs/organizationvalidation-sha-2-w-root.ca-bundle"},
{certfile, "/etc/pki/tls/certs/rabbitmq.pem"},
{keyfile, "/etc/pki/tls/private/rabbitmq.key"}
]}
]}
]}
].
% EOF
The issue came down to the value I specified for verify within the ssl_options config block. I used true when I should have used verify_peer.
The rabbitmq-server simply crashed and provided the above error. Once I corrected the config property things work fine so far.
I'm running on rhel7 with the rabbitmq.com centos/rhel7 package. rabbitmq-server-3.6.1-1.noarch and erlang also provided by rabbitmq.com erlang-18.3-1.el7.centos.x86_64
bilsch
All 16 comments
This is partially addressed by #550 but we should look into improving things for the classic config format.
michaelklishin
on 7 Apr 2016
Some more things that should be validated:
- Server certificate has proper "extended key usage" field
- Certificates/keys should be in PEM-format
- Cipher list should contain at least one cipher suitable for key type (EC, DSA, ...)
Because at runtime this errors will be reported as insufficient_security/handshake_failure/bad_certificate without any useful details.
binarin
on 11 May 2016
I had an interesting problem with the TLS/SSL options. For some reason it seems that RabbitMQ/Erlang does not verify that the specified cacertfile actually exists. I actually removed my cacert and restarted. The server was listening on the TLS port 5671, but didn't talk back to clients. No log entries were made with the default log levels. I'm running RabbitMQ 3.6.5 with Erlang 19.
Hydrochoerus
on 5 Oct 2016
In 3.7.0 it will for things we know are file paths.
On 5 Oct 2016, at 10:32, Hydrochoerus [email protected] wrote:
I had an interesting problem with the TLS/SSL options. For some reason it seems that RabbitMQ/Erlang does not verify that the specified cacertfile actually exists. I actually removed my cacert and restarted. The server was listening on the TLS port 5671, but didn't talk back to clients. No log entries were made with the default log levels. I'm running RabbitMQ 3.6.5 with Erlang 19.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
michaelklishin
on 5 Oct 2016
@Hydrochoerus I had a similar problem with an incorrect certfile path and spent 2 days trying to figure out why it's not working.
mydiemho
on 8 Nov 2016
@essen is improvements such as these something you'd consider (or already included into) recent/future Ranch releases?
michaelklishin
on 30 Nov 2016
Yes I am open to adding checks to catch errors early/provide better messages. 1.3.0 improved eaddrinuse and another but not the ones mentioned in this ticket. If you have more cases I'll be happy to add further checks.
essen
on 30 Nov 2016
I just spent one work day trying to figure out why amqps didn't work. It kept getting stuck trying to establish connection.
The keyfile had the wrong permissions (which probably caused some empty string to be used). It was my fault but it took me far too long to find out (testing with openssl s_server was fine since I had permissions).
This could have been avoided by a proper error in the logs. Server probably shouldn't even start in that case.
Algent
on 19 Jan 2017
@Algent 3.7.0 will validate file existence and readability for certain values that are known to be files. You can try it today.
Unreadable files do not cause empty strings to be used, they simply cause exceptions when a TCP connection upgrade to TLS is performed.
I'm not convinced that nodes should not even start if a file cannot be read but that's how it's going to work with 3.7.0, for better or worse.
michaelklishin
on 19 Jan 2017
I'm growing convinced that this is something that can be contributed upstream to Ranch. @essen, would you accept such a feature?
As for 3.7.0, the new config format already performs validation for file path values, so some of the most common cases are already covered. We can potentially do even more checks (e.g. that certificates can be loaded/are in the expected format) but there's risk to trying to be too smart with this stuff.
michaelklishin
on 9 Feb 2017
Can you show me what's the corresponding code in 3.7.0?
essen
on 9 Feb 2017
@essen it's not in RabbitMQ itself. Cuttlefish checks that file paths exist and are readable for values of a certain data type.
michaelklishin
on 9 Feb 2017
@michaelklishin I followed steps on https://www.rabbitmq.com/ssl.html but rabbit crashed on init
Here is the log. It doesn't stat what is wrong with the pem file. Can you help me find the issue?
I'm using v3.6.9 on ubuntu 16.04 BTW
=SUPERVISOR REPORT==== 1-Jun-2017::01:40:39 ===
Supervisor: {<0.262.0>,tcp_listener_sup}
Context: start_error
Reason: {shutdown,
{failed_to_start_child,ranch_acceptors_sup,
{listen_error,
{acceptor,{0,0,0,0,0,0,0,0},5671},
{options,{certfile,'/tmp/server/cert.pem'}}}}}
Offender: [{pid,undefined},
{id,{ranch_listener_sup,{acceptor,{0,0,0,0,0,0,0,0},5671}}},
{mfargs,
{ranch_listener_sup,start_link,
[{acceptor,{0,0,0,0,0,0,0,0},5671},
1,ranch_ssl,
[{port,5671},
{ip,{0,0,0,0,0,0,0,0}},
{max_connections,infinity},
{ack_timeout,5000},
{connection_type,supervisor},
inet6,
{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false},
{versions,['tlsv1.2','tlsv1.1',tlsv1]},
{cacertfile,'/tmp/testca/cacert.pem'},
{certfile,'/tmp/server/cert.pem'},
{keyfile,'/tmp/server/key.pem'},
{verify,verify_peer},
{fail_if_no_peer_cert,false}],
rabbit_connection_sup,[]]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]
=CRASH REPORT==== 1-Jun-2017::01:40:39 ===
crasher:
initial call: application_master:init/4
pid: <0.144.0>
registered_name: []
exception exit: {bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{{case_clause,
{error,
{{shutdown,
{failed_to_start_child,
{ranch_listener_sup,
{acceptor,{0,0,0,0,0,0,0,0},5671}},
{shutdown,
{failed_to_start_child,ranch_acceptors_sup,
{listen_error,
{acceptor,{0,0,0,0,0,0,0,0},5671},
{options,
{certfile,'/tmp/server/cert.pem'}}}}}}},
{child,undefined,
'rabbit_tcp_listener_sup_:::5671',
{tcp_listener_sup,start_link,
[{0,0,0,0,0,0,0,0},
5671,ranch_ssl,
[inet6,
{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false},
{versions,['tlsv1.2','tlsv1.1',tlsv1]},
{cacertfile,'/tmp/testca/cacert.pem'},
{certfile,'/tmp/server/cert.pem'},
{keyfile,'/tmp/server/key.pem'},
{verify,verify_peer},
{fail_if_no_peer_cert,false}],
rabbit_connection_sup,[],
{rabbit_networking,tcp_listener_started,
['amqp/ssl',
[{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false},
{versions,['tlsv1.2','tlsv1.1',tlsv1]},
{cacertfile,'/tmp/testca/cacert.pem'},
{certfile,'/tmp/server/cert.pem'},
{keyfile,'/tmp/server/key.pem'},
{verify,verify_peer},
{fail_if_no_peer_cert,false}]]},
{rabbit_networking,tcp_listener_stopped,
['amqp/ssl',
[{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false},
{versions,['tlsv1.2','tlsv1.1',tlsv1]},
{cacertfile,'/tmp/testca/cacert.pem'},
{certfile,'/tmp/server/cert.pem'},
{keyfile,'/tmp/server/key.pem'},
{verify,verify_peer},
{fail_if_no_peer_cert,false}]]},
1,"SSL Listener"]},
transient,infinity,supervisor,
[tcp_listener_sup]}}}},
[{rabbit_networking,start_listener0,5,
[{file,"src/rabbit_networking.erl"},{line,294}]},
{rabbit_networking,'-start_listener/5-lc$^0/1-0-',5,
[{file,"src/rabbit_networking.erl"},{line,282}]},
{rabbit_networking,start_listener,5,
[{file,"src/rabbit_networking.erl"},{line,282}]},
{rabbit_networking,'-boot_ssl/1-lc$^0/1-0-',3,
[{file,"src/rabbit_networking.erl"},{line,142}]},
{rabbit_networking,boot_ssl,1,
[{file,"src/rabbit_networking.erl"},{line,142}]},
{rabbit_networking,boot,0,
[{file,"src/rabbit_networking.erl"},{line,128}]},
{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,
[{file,"src/rabbit_boot_steps.erl"},{line,49}]},
{rabbit_boot_steps,run_step,2,
[{file,"src/rabbit_boot_steps.erl"},{line,49}]}]}}}}
in function application_master:init/4 (application_master.erl, line 134)
ancestors: [<0.143.0>]
messages: [{'EXIT',<0.145.0>,normal}]
links: [<0.143.0>,<0.7.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 2586
stack_size: 27
reductions: 290
neighbours:
and this is my /tmp/server folder
-rw-rw-r-- 1 llu llu 1070 Jun 1 01:35 cert.pem
-rw-rw-r-- 1 llu llu 0 Jun 1 01:16 index.txt
-rw-rw-r-- 1 llu llu 2349 Jun 1 01:36 keycert.p12
-rw-rw-r-- 1 llu llu 1675 Jun 1 01:34 key.pem
-rw-rw-r-- 1 llu llu 924 Jun 1 01:34 req.pem
evollu
on 1 Jun 2017
@evollu this is not a support venue. Please post your question to rabbitmq-users and provide full config file.
michaelklishin
on 1 Jun 2017
@evollu ok, looks like you've used single quotes instead of double quotes for a file path: {certfile,'/tmp/server/cert.pem'}. The config file is an Erlang term file and single quotes in Erlang do not produce a string. Use double quotes.
michaelklishin
on 1 Jun 2017
Oh stupid mistake. I'm new to erlang. It is working now!
Thanks for your timely help 👍
evollu
on 1 Jun 2017
Related issues
gerhard
·
8Comments
yebangyu
·
4Comments
lukebakken
·
5Comments
lukebakken
·
7Comments
ppyoosuf
·
5Comments
Most helpful comment
I had an interesting problem with the TLS/SSL options. For some reason it seems that RabbitMQ/Erlang does not verify that the specified cacertfile actually exists. I actually removed my cacert and restarted. The server was listening on the TLS port 5671, but didn't talk back to clients. No log entries were made with the default log levels. I'm running RabbitMQ 3.6.5 with Erlang 19.