Elasticsearch: Enhance Help for plugin authors page

[gmoskovicz]

This page is often confusing for the new 2.x release. Some links don't work (EG: https://github.com/elastic/elasticsearch/blob/master/dev-tools/src/main/resources/plugin-metadata/plugin-descriptor.properties , and the information on how to create plugins is not very clear.

It will be great to have a better description on how to create plugins.

[clintongormley]

I've fixed the URL for the plugin-descriptor file in 2.x, and updated this page for gradle in master.

@rjernst you wanted to provide more info for plugin authors?

[rjernst]

@clintongormley I will jot my notes about info for plugin authors here. I'm not sure what the best place for them is, but I figure you will know.

In gradle, users do the equivalent of the shared pom by importing the elasticsearch.esplugin gradle plugin. In a build.gradle file, that looks something like this:

buildscript {
  repositories {
    maven {
      name 'sonatype-snapshots'
      url "https://oss.sonatype.org/content/repositories/snapshots/"
    }
  }
  dependencies {
    classpath "org.elasticsearch.gradle:build-tools:3.0.0-snapshot"
  }
}
apply plugin: 'elasticsearch.esplugin'

Note that the maven part should be changed to mavenCentral() once we've actually released master.

Once a plugin author has applied this gradle plugin to their build, they get a number of builtin tasks, as well as configuration options.

Configuration

Configuration of the ES plugin happens using the esplugin extension. A full configured esplugin with all options specified would look like the following:

esplugin {
  name 'my-plugin' // required, the name of the plugin which will be used as a unique identifier within ES
  version '3.0.0' // optional, this is an arbitrary version number for your plugin, and defaults to the version of elasticsearch
  description 'My plugin does foo and bar' // required, a short description of what the plugin does
  classname 'com.foo.MyPlugin' // required, the full qualified class name that implements o.e.plugins.Plugin
}

Tasks

In addition to simply building the plugin, a number of tasks are added, which are the same as Elasticsearch itself uses to check the software.

precommit

These tasks do static checks on the plugin code. They do things like check for jarhell (before actually running tests, rather than get crazy errors in the tests themselves), check for forbidden patterns like nocommit, run forbidden apis, etc. (We can list these out explicitly along with better descriptions, but these can and will change in the future so the list may get out of sync.

test

These are unit tests, run with the same randomized runner that Lucene and ES tests are based on. All test classes should end with the name "Tests".

integTest

These are REST integration tests. They spin up an ES node, and run rest tests against it. The yaml files for the rest tests should be in src/test/resources/rest-api-spec/test.

I'm sure I'm missing something, but that is a simple breakdown of what esplugin provides.

[clintongormley]

@rjernst could you also provide a few common commands, eg how do you build, test, package, how do you run the precommit task? how do you deploy?

[rjernst]

I'm not sure we should write a gradle tutorial...there are plenty out there.

For deploying to nexus, they can add the following to their file, and the esplugin is already set up to generate a correct pom:

apply plugin: 'com.bmuschko.nexus'

All 3 of the tasks i mentioned (precommit, test and integTest) are run as part of check, which is a standard gradle task you get out of the box.

I also forgot to mention before some additional things:

• Transitive dependencies are not allowed, at least in the standard compile and testCompile configurations.
• Standard repositories are added automatically by applying the elasticsearch.esplugin plugin. These are maven central, sonatype snapshots, and the lucene snapshot repo for snapshot builds of ES.
• javac is configured with the compact3 profile by default (can be changed by setting the compactProfile setting in the project). It is also set to run doclint.
• The jar built for the plugin automatically includes information about the version of ES, lucene and java used to build it, as well as repository information (if git is used).
• A licenses directory is expected with licenses for each dependency, and shas. The shas can be updated using gradle updateShas, and the task can be disabled with dependencyLicenses.enabled = false.

[villasv]

I came across this same issue https://github.com/spinscale/elasticsearch-ingest-opennlp/issues/8 here with gradle 2.10 (from ubuntu apt). Are there still gradle version limitations for the esplugin?

[nik9000]

Are there still gradle version limitations for the esplugin?

Yes. We've got verbal assurance from the gradle folks that they'll fix the issue on their side so we're just waiting.... For now 2.13 is required. No less and no more, sadly.

[stephenprater]

This is weird, since you can't actually even get Gradle 2.13 from the gradle distribution site anymore.

Edit - You can get it here.

  url "https://services.gradle.org/distributions/gradle-2.13-bin.zip"
  sha256 "0f665ec6a5a67865faf7ba0d825afb19c26705ea0597cec80dd191b0f2cbb664"

[dadoonet]

What I did recently is to create an empty Gradle project and run

gradle wrapper --gradle-version 2.13

to get a wrapper. Then I copied files to elasticsearch dir.

HTH

[elasticmachine]

Pinging @elastic/es-core-infra

[debadair]

[docs issue triage]

Not sure about the specifics, but this is still at least partially relevant.

[williamrandolph]

There are no broken links on the help for plugin authors page, but it's still fairly sparse and could be improved.

[williamrandolph]

This comment had a good to-do list for improving this documentation page:

https://github.com/elastic/elasticsearch/issues/41494#issue-436747396

I closed that ticket as a duplicate.