Elasticsearch: Unable to remove old plugins in ES 6.2.0 with elasticsearch-plugin

[holgr]

Elasticsearch version:
Version: 6.2.0, Build: 37cdac1/2018-02-01T17:31:12.527918Z, JVM: 1.8.0_161

Plugins installed:
[ingest-geoip, ingest-user-agent]

JVM version:
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

OS version:
3.10.0-693.17.1.el7.x86_64 (CentOS 7.4.1708)

Description of the problem including expected versus actual behavior:
Uninstalling an old plugin fails after upgrading to 6.2.0. Currently using elasticsearch-plugin I'm unable to remove old plugins in order to install updated plugins that are compatible with 6.2.0.

Steps to reproduce:

1. Get list of installed plugins with ./elasticsearch-plugin list
   ingest-geoip
   WARNING: plugin [ingest-geoip] is incompatible with version [6.2.0]; was designed for version [6.1.3]
   ingest-user-agent
   WARNING: plugin [ingest-user-agent] is incompatible with version [6.2.0]; was designed for version [6.1.3]
2. Run ./elasticsearch-plugin remove ingest-geoip
   Exception in thread "main" java.lang.IllegalArgumentException: plugin [ingest-geoip] is incompatible with version [6.2.0]; was designed for version [6.1.3]
   at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:224)
   at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:329)
   at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:89)
   at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:67)
   at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
   at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
   at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:75)
   at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
   at org.elasticsearch.cli.Command.main(Command.java:90)
   at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:48)

1. Similar results with ./elasticsearch-plugin remove ingest-user-agent
   Exception in thread "main" java.lang.IllegalArgumentException: plugin [ingest-geoip] is incompatible with version [6.2.0]; was designed for version [6.1.3]
   at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:224)
   at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:329)
   at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:89)
   at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:67)
   at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
   at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
   at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:75)
   at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
   at org.elasticsearch.cli.Command.main(Command.java:90)
   at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:48)

Provide logs (if relevant):

See above.

[jasontedor]

This reproduces. Thanks for the report.

As a workaround, you can downgrade to 6.1.3, remove the plugins, upgrade, and reinstall your plugins. I know this is terrible, but this is all I can offer right now until we have a release with a fix in place.

We will work on having a fix out as soon as possible.

[holgr]

@jasontedor Thanks for the quick response! :) Alternatively I should be able to remove them from $PLUGIN_PATH and then just install as usual?

[jasontedor]

My preference would be the downgrade/use the official tool/re-upgrade route; I do not want to encourage messing with the underlying files (e.g., some plugins add files in other locations, e.g., bin).

[jasontedor]

I have opened #28540.

[nin9s]

@jasontedor could you please provide the steps to downgrade 6.2 to 6.1.3 please. Until now I thought it isnt possible to downgrade elasticsearch

[jasontedor]

@strrrn Typically you can not. However, in the context of this bug Elasticsearch will not have successfully started (because an old plugin prevents that). This means that no changes would be made to files on disk (e.g., the cluster state) so the previously-installed version will still be able to read these files. This is why downgrading is safe here while typically it is not.

Regarding how to downgrade, I can not answer this because I do not know how you install Elasticsearch, nor how you upgrade Elasticsearch. This is not the place to have that discussion though. Instead, please use the forum. You can feel free to link to a topic that you open there from here and I will follow that link and try to help you.

[nin9s]

@jasontedor thank you very much and for explaining the reason in detail!
The thread is located here https://discuss.elastic.co/t/how-to-downgrade-es-from-6-2-to-6-1-3/118904

[jasontedor]

We shipped Elasticsearch 6.2.1 today with a fix for this issue.

[stevebanik-ndsc]

Any idea why the "list" flag seemingly attempts to install the repository-s3 plugin?

```[root@ip-10-174-7-130 elasticsearch]# /usr/share/elasticsearch/bin/elasticsearch-plugin list --verbose
Plugins directory: /usr/share/elasticsearch/plugins
.installing-7024295464285032320

• Plugin information:
  Name: repository-s3
  Description: The S3 repository plugin adds S3 repositories
  Version: 6.5.4
  Elasticsearch Version: 6.5.4
  Java Version: 1.8
  Native Controller: false
  Extended Plugins: []
  
  
  
  • Classname: org.elasticsearch.repositories.s3.S3RepositoryPlugin```
  
  

I was unable to start ES 6.6.0, was unable to remove repository-s3, had to downgrade to ES 6.5.4 in order to get ES started.

[jasontedor]

That looks like a temporary directory from a failed plugin install. List dies not reach out to the Internet and install plugins. We should probably fail list if leftovers like tat are on disk.