Logstash: Plugin doc gen script pulls wrong text for plugin docs

[dedemorton]

The script for generating the plugin docs is currently pulling in a couple of code comments instead of the description that begins at https://github.com/logstash-plugins/logstash-filter-geoip/blob/master/lib/logstash/filters/geoip.rb#L34.

Here's what it looks like in the doc (including a screen capture because Suyog might fix the doc before you see the problem).

The script should be resilient enough to extract the text even if there's a minor formatting snafu (like an extra line).

[dedemorton]

@ph I'm seeing a similar problem here where the script takes text that's meant to be a comment: https://www.elastic.co/guide/en/logstash/5.2/plugins-inputs-http.html

[dedemorton]

Here's another one. But in this case, there is no description of what the plugin does. "handles disconnects" comes from a code comment, tho. :-(

[dedemorton]

Here's another one.

[dedemorton]

From https://www.elastic.co/guide/en/logstash/5.2/plugins-outputs-boundary.html, not sure if this is a problem with the doc gen or the code itself:

[dedemorton]

This one is pulling the license agreement and squishing it together with the description:

same problem exists in https://www.elastic.co/guide/en/logstash/5.2/plugins-outputs-google_cloud_storage.html

[dedemorton]

[dedemorton]

I'm guessing this is not the droid we're looking for:

[dedemorton]

[dedemorton]

[dedemorton]

[dedemorton]

[dedemorton]

[dedemorton]

[dedemorton]

[dedemorton]

@ph Including screen captures here for all the issues I found even tho they are likely related to the same bug. I just want to be able to verify the fix when you're done.

[dedemorton]

alvin found this one, too: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html

[ph]

thanks @dedemorton checking it right now since we will push a few new artifacts.

[ph]

Got it reproduced, https://github.com/logstash-plugins/logstash-codec-msgpack/blob/master/lib/logstash/codecs/msgpack.rb#L20, It seems that on some file the parser doesn't known it should have stop reading the headers.

[ph]

Fixed:

• [x] logstash-input-http
• [x] logstash-filter-geoip
• [x] logstash-filter-msgpack
• [x] logstash-output-http
• [x] logstash-codec-collectd
• [x] logstash-filter-fingerprint
• [x] logstash-filter-elapsed -> NEED PR to remove lines before the requires https://github.com/logstash-plugins/logstash-filter-elapsed/blob/master/lib/logstash/filters/elapsed.rb#L1
• [x] logstash-output-sqs -> NEED PR to remove comments before https://github.com/logstash-plugins/logstash-output-sqs/blob/master/lib/logstash/outputs/sqs.rb#L10-L13
• [x] logstash-output-rabbitmq -> Need a another patch in the generator for the module structure.
• [x] logstash-output-mongodb -> Need a PR to remove https://github.com/logstash-plugins/logstash-output-mongodb/blob/master/lib/logstash/outputs/mongodb.rb#L7
• [x] logstash-output-loggly
• [x] logstash-output-jira -> Need a formating PR for the example code.
• [x] logstash-output-google_bigquery -> need a formating PR for the license https://github.com/logstash-plugins/logstash-output-google_bigquery/blob/master/lib/logstash/outputs/google_bigquery.rb
• [x] logstash-output-google_cloud_storage -> need a formating PR for the license https://github.com/logstash-plugins/logstash-output-google_cloud_storage/blob/master/lib/logstash/outputs/google_cloud_storage.rb
• [x] logstash-output-boundary
• [x] logstash-input-xmpp -> Will need a PR to provide a description
• [x] logstash-input-stomp -> Will need a PR to provide a description

[acchen97]

[ph]

@dedemorton For the license in the file, I think the license content should go in the LICENSE file in the plugin repo? I think this is where it should belongs.

[ph]

@dedemorton I've merged the PRs for the plugins and published the plugins.

[acchen97]

@ph will this fix only be executed on new doc versions? Or can we also fix older versions retroactively?

[ph]

@acchen97 it depends, by looking at the list at https://github.com/elastic/logstash/issues/6692#issuecomment-279849370, if they make no mention of a PR we could fix older versions with the same script.

[dedemorton]

@ph I think we should fix the older versions. For the topics that can't be fixed just by running the docgen tool again, I could tweak the asciidoc, but I only want to do that if we're sure we won't have to run the tool against those versions again.

[ph]

@dedemorton Exactly, I would have preferred to not have to do it manually. :(

[ph]

@dedemorton dont worry about the google plugins I have fixed it in the parser.

[ph]

@dedemorton I've merged the changes to the logstash-docgen tool so on the next generation a lot of theses issues should be fixed.

I will assign this issue to you for the manual changes.

[dedemorton]

Closing because the manual changes have been made, where feasible, to the source files.