Elasticsearch: Secure Settings

[rjernst]

This is a meta issue to track work on making sensitive settings secure in elasticsearch. The core infrastructure for this feature is added in #22335, which provides the elasticsearch-keystore tool. The following work is still necessary to consider the feature complete:

• [x] Convert existing settings (eg aws keys) to new infrastructure
• [x] Add password support to the keystore
• [x] Add reading password for start scripts (init.d and systemd)
• [x] Investigate best location for keystore file. Currently in config dir, which has read permissions for the life of the node, but could place somewhere with no SM permissions, since reading is done before SM is setup.

The following would be nice to haves:

• [ ] Investigate only allowing reading secure settings registered by plugins (eg not allowing reading other plugin's secure settings)
• [x ] Investigate setting explicit algorithm for PBE
• [ ] Add support for private keys
• [ ] Add support for certificates
• [ ] Add support to read secure settings from Vault instead of keystore

[joshbressers]

I filed an issue to cover the specific feature of password protecting the keystore
https://github.com/elastic/elasticsearch/issues/32691

[pdanysz]

Dears,
I faced an issue with elasticsearch.keystore location when using Elastic on Kubernetes clusters and configuration are mounted via ConfigMaps. Due to this I have "read-only" fs of /etc/elasticsearch and because of that , ES is going to crash.

Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.FileSystemException: /etc/elasticsearch/elasticsearch.keystore.tmp: Read-only file system
--
  | Likely root cause: java.nio.file.FileSystemException: /etc/elasticsearch/elasticsearch.keystore.tmp: Read-only file system
  | at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
  | at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
  | at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
  | at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
  | at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
  | at java.nio.file.Files.newOutputStream(Files.java:216)
  | at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:411)
  | at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:407)
  | at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:255)
  | at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:467)
  | at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:238)
  | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:295)
  | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
  | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
  | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
  | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
  | at org.elasticsearch.cli.Command.main(Command.java:90)
  | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116)
  | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
  | Refer to the log for complete error details.

It will be great to have a settings for keystore location in elasticsearch.yaml to set directory outside of /etc/elasticsearch.

Can you advice in this topic?

Thanks in advance,

[gwbrown]

I'm labeling this team-discuss to determine which of the as-yet-incomplete "nice to have" tasks we still intend to complete. Once we do so, they should be promoted to individual issues and this issue closed, as the main thrust of this issue has been long since completed.