Elasticsearch: certutil cert: using shell metacharacters in passwords

Created on 6 Jun 2018 · 10Comments · Source: elastic/elasticsearch

Elasticsearch version (bin/elasticsearch --version):

Version: 6.2.4, Build: ccec39f/2018-04-12T20:37:28.497551Z, JVM: 1.8.0_171

Plugins installed: [ x-pack ]

Description of the problem including expected versus actual behavior:
It might be common to use shell metacharacters like $, `, \, etc. in passwords supplied in the --pass flag of certutil cert. E.g.,

./certutil cert --pass pas$word -out test.p12

This may not yield the expected result since $ will be interpreted by the shell. If $word is unset, then the password is actually pas:

openssl pkcs12 -in test.p12 -noout -passin pass:pas
MAC verified OK

In other words,

word=sword
./certutil cert --pass pas$word -out test.p12
openssl pkcs12 -in test.p12 -noout -passin pass:password
MAC verified OK

But wait! Prior to setting $word, it _seemed_ to work fine:

/usr/share/elasticsearch/bin/x-pack# openssl pkcs12 -in test.p12 -noout -passin pass:pas$word
MAC verified OK

Actually no, because openssl also interprets $. Since $word is not set, then the value passed is actually pas.

I would like to document this behavior in the certutil docs and how to avoid it by escaping desired metacharacters. For example, this will yield something more expected:

./certutil cert --pass pas\$word -out test.p12
openssl pkcs12 -in test.p12 -noout -passin pass:pas\$word
MAC verified OK

Please note, omitting the --pass flag and supplying a password to certutil when prompted avoids the issue entirely.

:SecuritNetwork >docs discuss

Source

inqueue

Most helpful comment

To me this feels like we would be documenting basic shell conventions that could lead us into having to document more; for example there is windows and it’s characters.

I completely agree with this and I too think that we should avoid this.

Perhaps we could add just a very short one line statement about it?

I would rather that we didn't. Every one of these carries a weight on the reader and on the maintainers, and it's hard to cover them adequately.

jasontedor on 7 Jun 2018

👍3

All 10 comments

Pinging @elastic/es-security

elasticmachine on 6 Jun 2018

To me this feels like we would be documenting basic shell conventions that could lead us into having to document more; for example there is windows and it’s characters.

Does OpenSSL document anything about shell characters?

jaymode on 6 Jun 2018

Does OpenSSL document anything about shell characters?

Good point, not that I have ever seen and I am certainly not out to force extra work on the docs team. Perhaps we could add just a very short one line statement about it?

inqueue on 6 Jun 2018

Sorry, I misread the OP, don't mind me. Comment removed.

DaveCTurner on 6 Jun 2018

To me this feels like we would be documenting basic shell conventions that could lead us into having to document more; for example there is windows and it’s characters.

I completely agree with this and I too think that we should avoid this.

Perhaps we could add just a very short one line statement about it?

I would rather that we didn't. Every one of these carries a weight on the reader and on the maintainers, and it's hard to cover them adequately.

jasontedor on 7 Jun 2018

👍3

I have one last note before I close it. The scenario I would like to help users avoid is having an incorrect password on keystores created by certutil. Setting up TLS can be a very involved process, even with great tools like certuil to help.

Please consider:

Someone new to X-Pack is setting up security.
They have read the X-Pack documentation for setting up TLS between cluster nodes.
The user follows all directions precisely:
- generate node certificates using certutil for every node. At this point, the user has read and understood the documentation for certutil, written an instances.yml and is off to the races.
- copy the generated P12 files to each node in the cluster
- configure elasticsearch.yml on every node
- manually add P12 keystore passwords to elasticsearch.keystore with elasticsearch-keystore on every node. This also is a new concept and an excellent security measure btw. The vigilant user reads Secure Settings documentation to learn about it.
The user restarts Elasticsearch which results in exiting to error.

Incorrect Keystore Password Errors

[2018-06-05T12:22:19,579][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [master-01] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:563) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:505) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:422) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:146) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.node.Node.<init>(Node.java:303) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_161]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:554) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:505) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:422) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:146) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.node.Node.<init>(Node.java:303) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:72) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:419) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:455) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:91) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:127) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_161]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:554) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:505) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:422) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:146) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.node.Node.<init>(Node.java:303) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 6 more
Caused by: java.io.IOException: keystore password was incorrect
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059) ~[?:?]
        at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_161]
        at org.elasticsearch.xpack.core.ssl.CertUtils.readKeyStore(CertUtils.java:276) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.CertUtils.trustManager(CertUtils.java:267) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:70) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:419) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:455) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:91) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:127) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_161]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:554) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:505) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:422) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:146) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.node.Node.<init>(Node.java:303) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 6 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059) ~[?:?]
        at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_161]
        at org.elasticsearch.xpack.core.ssl.CertUtils.readKeyStore(CertUtils.java:276) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.CertUtils.trustManager(CertUtils.java:267) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:70) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:419) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:455) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:91) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:127) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_161]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:554) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:505) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:422) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:146) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.node.Node.<init>(Node.java:303) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 6 more

Go back to step 3 and do it again only to receive the same result because there is an unescaped shell metacharacter in the --pass flag.

I am in full agreement that we are not in a position to educate users on the fundamentals of shell usage. That is a trap we do not want to fall into. So instead of documenting how to use shell metacharacters in the --pass flag, maybe we could provide a suggestion in the documentation on how to verify passwords set on P12 files or provide a method in certutil to do so. The former could just be openssl pkcs12 without the -passin option which will prompt the user for the keystore password:

openssl pkcs12 -in master01.p12 -info
Enter Import Password:

inqueue on 7 Jun 2018

We discussed this in Fix-it-(Thursday|Friday) and the team is in agreement that we should not make any documentation changes here to bring in fundamentals that are outside of our products.

jasontedor on 7 Jun 2018

The user follows all directions precisely:

Our directions _never_ instruct them to use the --pass option, so if they follow the directions precisely they will never run into this problem.

tvernum on 8 Jun 2018

Our directions never instruct them to use the --pass option, so if they follow the directions precisely they will never run into this problem.

This is true; the docs do not state to use the --pass option. If a user wants to just enable authentication and/or enable TLS, at a minimum self-signed certificates must be generated in order to get through the rest of the setup process. My ask here was for a helpful tip in the docs to save X-Pack users some time in a process that can take a considerable amount of work in reading docs and setup to complete.

Will try to solve this in other ways that do not include public documentation enhancements.

inqueue on 8 Jun 2018

Part of the issue here is where do we stop?

One of the reasons why the docs don't instruct customers to use --pass is that it can be a security issue. Those commandline parameters are visible in ps, so that password is exposed to other local users on the same box. If we're going to take responsibility to explain to customers how their OS/Shell works, then we ought to also document the potential security issues in every tool that has the option to accept a password in the commandline.

To understand why trying to briefly explain shell quoting is hard try reading the man page for various versions of the unix find utility. The -exec option has special interpretation for {} and ;, but the shell may also interpret those in special ways, so in many cases the authors of those man pages have tried to call that out for readers. And none of them do a good job, despite the fact that they're only calling out 2 very specific syntaxs.

GNU Find