Logstash: Docs generation, v2.

[ph]

Motivation:

Our current documentation generation has currently a few issues that we want to address in Logstash 5.0.

• No easy way when you are a plugin author to visualize what would be the doc output.
• No easy way to make sure you doc will correctly build in the general process (ie: when referencing other parts of the doc?)
• Docs can be _out of sync_ from a plugin release vs a Logstash release.
• Using mixins can confuse the docs generation and can make it miss options.
• Won't work with pure java plugins (NOT A SHORT TERM PRIORITY)

TODO

• Make sure all the community plugins have the community header.
• Move docs generations to tools/docs or external so PR can use it.
• Show plugin version in the generated doc
• On merge publish
• docs build should be part of the PR check process
• Generating the whole doc should be a Jenkins job
• refactor the parser into a smaller and testable unit.
  
  
  
  • split parsing or the rendering
  
  
  • split the file list from the provided files
  
  

[ph]

I am currently evaluating the current code and see if we could make it better/more flexible.

[ph]

OK, I have refactored the generation of the doc a bit and I merged the work I've done in a previous PR.

I have theses feature:

• in the plugin bundle exec rake doc:asciidoc
• in the plugin bundle exec rake doc:html
• logstash-plugin help my-plugin

I am still not happy with the file structure and I need to add test for it.

[ph]

After talking with @suyograo on zoom, we agreed that the logstash-plugin help wasn't really needed and I will remove this feature to add better separation of the tooling from the core.

[ph]

Also we agreed to create a new gem defined in the logstash repo named logstash-docgen which will give you the tools needed to build all the documentation.

[ph]

The gem will be able to do the following:

Generate the doc from all the plugin
Generate the doc for a specific plugin
Generate the doc for a specific plugin branch.

[ph]

It seems some of the generated documentation for the plugins doesn't include the mention of the community supported plugins.

logstash-output-email is not in this list https://github.com/elastic/logstash/blob/master/rakelib/default_plugins.rb and doesn't have the header.

[ph]

I have created a tools thats does the following:

Allow plugin developers to generate the documentation with the following commands:

bundle exec rake doc:asciidoc
bundle exec rake doc:asciidoc:html # open a browser with the htmlified asciidoc.

A CLI tool that allow to generate the documentation for a specific plugins, a list or all the plugins from logstash-plugins.

logstash-docgen/bin/logstash-docgen logstash-input-http
logstash-docgen/bin/logstash-docgen -a --target output

[ph]

The new CLI tools has a neat error handling.

~/e/l/logstash-docgen git:master ❯❯❯ rm -rf target;bin/logstash-docgen --sample                                                                                                                                                                                                                                           ✱ ◼
Processing, 15 plugins: logstash-input-sqs, logstash-input-kafka, logstash-input-file, logstash-codec-cef, logstash-filter-cidr, logstash-filter-elasticsearch, logstash-filter-translate, logstash-filter-punct, logstash-output-kafka, logstash-input-github, logstash-filter-anonymize, logstash-codec-cloudtrail, logstash-codec-collectd, logstash-codec-compress_spooler, logstash-codec-dots
    logstash-input-sqs > SUCCESS
    logstash-input-kafka > SUCCESS
    logstash-input-file > SUCCESS
    logstash-codec-cef > SUCCESS
    logstash-filter-cidr > SUCCESS
    logstash-filter-elasticsearch > SUCCESS
    logstash-filter-translate > SUCCESS
    logstash-filter-punct > SUCCESS
    logstash-output-kafka > SUCCESS
    logstash-input-github > SUCCESS
    logstash-filter-anonymize > SUCCESS
    logstash-codec-cloudtrail > SUCCESS
    logstash-codec-collectd > SUCCESS
    logstash-codec-compress_spooler > SUCCESS
    logstash-codec-dots > SUCCESS
Docs generated in 402.168 seconds

If something goes wrong with any of the jobs, it will mark the job as FAIL and render all the errors at the end of the run. But if you interrupt the process right away it will output the current errors.

[geekpete]

While this tool is being refactored, can I ask for a feature to be able to generate documentation glossaries?

What I want to be able to get out of source code is:

• a definitive list of every configuration setting including defaults, possible range/values, context (what that setting effects and where it can be used) and description with links to specific doco where applicable.
• a definitive list of every stats api metric including possible ranges, context (what components that metric can apply to), what unit the metric is measured in and description with links to specific doco where applicable.

I want the ability to automatically produce a configuration glossary and a stats/metrics glossary directly from source code, to be included as a standard part of the documentation.

The way I see this working is everywhere a configuration setting/variable or a stats metric is first defined in the source code, there should also be associated inline documentation to go with it that can be picked up by the glossary generator. It would have to include which glossary it would/could be part of.

Until we get this built for all Elastic products, I'm going to be building glossaries by hand, starting with the Elasticsearch stats, just because of how useful these glossaries can be.

[ph]

a definitive list of every configuration setting including defaults, possible range/values, context (what that setting effects and where it can be used) and description with links to specific doco where applicable.

In the doc we already list all of the possible options for a plugins? What would be different with your proposal? Can you clarify it a bit maybe with a plugin context?

a definitive list of every stats api metric including possible ranges, context (what components that metric can apply to), what unit the metric is measured in and description with links to specific doco where applicable.

This is a great idea and also the perfect time to bring it to the table we are in the process of adding theses metrics but we didn't discuss anything related to their documentation.

@purbon @jordansissel Concerning the documentation parser, I believe its relatively easy to parse metric calls in the plugins.

I see two possible solution, either we capture comments before the metric calls or we add an hash arguments to every metric calls.

Option A:

# The number of concurrent connection.
metric.increment(:connections)

Option B:

metric.increment(:connection, :doc => "Number of concurrent connection")

Option B is probably less errors prone.

WDYT?

[geekpete]

The configuration glossary idea was originally for Elasticsearch.
There are many tweakable settings/configs with ES so a single place to look up what any one of them is and how it can be used was my idea of something useful.

For configs, this might not translate over to Logstash as well.

[ph]

@geekpete Yeah, It make sense for ES, but in Logstash we have way to many different concepts I doubt they would make sense in a glossary. But in the other hand this could help to standardize some options over the plugins.

[ph]

I will close this issue since this was mostly a meta issue and we have create specific issue for the doc since then.