Logstash: 5.5.0 Task: Documentation build is not working correctly

[jordansissel]

% grep 'Fetch of' /tmp/x
Fetch of https://raw.githubusercontent.com/logstash-plugins/logstash-output-xmpp/v3.0.2/docs/index.asciidoc failed: #<Net::HTTPNotFound:0x0055f2a224be78>
Fetch of https://raw.githubusercontent.com/logstash-plugins/logstash-output-gelf/v3.1.2/docs/index.asciidoc failed: #<Net::HTTPNotFound:0x0055f2a26a9dd0>
Fetch of https://raw.githubusercontent.com/logstash-plugins/logstash-output-websocket/v3.0.1/docs/index.asciidoc failed: #<Net::HTTPNotFound:0x0055f2a25e9e90>
Fetch of https://raw.githubusercontent.com/logstash-plugins/logstash-filter-truncate/v1.0.0/docs/index.asciidoc failed: #<Net::HTTPNotFound:0x0055f2a2376d48>
Fetch of https://raw.githubusercontent.com/logstash-plugins/logstash-input-kinesis/v2.0.3/docs/index.asciidoc failed: #<Net::HTTPNotFound:0x0055f2a2375fd8>

At least 5 plugins are targeted for Logstash 5.5..0 but they have no docs/index.asciidoc which probably means either:

• maybe: They got docs migrated (docs/index.asciidoc) but did not get released
• or, maybe: they got released, but 5.5.0 cannot install them due to dependency conflicts
• or maybe: something else.

[jordansissel]

Currently blocked on a separate task. OOM while generating the plugins list --

% JAVA_OPTS="-Xmx1g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=." bin/bundle exec rake generate_plugins_version
WARNING: can not set Session#timeout=(300) no session context
WARNING: can not set Session#timeout=(300) no session context
Failed to install: logstash-filter-alter
WARNING: can not set Session#timeout=(300) no session context
java.lang.OutOfMemoryError: Java heap space
Dumping heap to ./java_pid11837.hprof ...
Heap dump file created [1342137595 bytes in 7.005 secs]
rake aborted!
Java::JavaLang::OutOfMemoryError: Java heap space
org.jruby.RubyArray.newArrayNoCopyLight(RubyArray.java:237)
org.jruby.Ruby.newArrayNoCopyLight(Ruby.java:3312)
org.jruby.runtime.BlockBody.newArgsArrayFromArgsWithoutUnbox(BlockBody.java:282)
org.jruby.runtime.CompiledBlock19.call(CompiledBlock19.java:87)
org.jruby.runtime.Block.call(Block.java:101)
org.jruby.RubyProc.call(RubyProc.java:300)
org.jruby.RubyProc.call19(RubyProc.java:281)

I have tried with -Xmx64g and it still OOMs.

[jordansissel]

• logstash-input-kinesis/v2.0.3

This is the latest version. Needs publishing

[jordansissel]

• logstash-filter-truncate/v1.0.0

Wrong version; expected latest of v1.0.2 (published June 23)

[jordansissel]

• logstash-output-websocket/v3.0.1

Wrong version; expected latest of 3.0.2

[jordansissel]

• logstash-output-gelf/v3.1.2

Wrong version. Expected v3.1.3

Hypothesis: Dependency resolution is preventing v3.1.3 from being installed.

Data:

• logstash-input-gelf has a development dependency on gelf gem v1.3.2.
• logstash-output-gelf v3.1.3 depends on gelf gem v3.0.0
• logstash-output-gelf v3.1.2 depends on gelf gem v1.3.2
• in Logstash 5.4.0, latest input-gelf and output-gelf are fine to install.

Conclusion based on this data: Somehow the all-plugins list tool is resolving dependencies with development dependencies enabled.

PR: https://github.com/logstash-plugins/logstash-input-gelf/pull/56

[jordansissel]

• logstash-output-xmpp/v3.0.2/

Wrong version. Expected v3.0.4

[jordansissel]

for posterity -- @ph has a fix for the OOM problem in testing right now.

[ph]

PRs needed to review to fix the xmpp issues.

https://github.com/logstash-plugins/logstash-output-xmpp/pull/6
https://github.com/logstash-plugins/logstash-input-xmpp/pull/10

Problems: incompatible dependencies on XMPP4R.

[ph]

OOM fix is now merged in the right branches, see: https://github.com/elastic/logstash/pull/7544

[ph]

I think https://rubygems.org/gems/logstash-output-websocket was published after the lock was generated, we didn't change anything on the dependencies side, It's the same as 3.0.1.

[ph]

Kinesis was missing the publish gem task I've created the following PR to add it, https://github.com/logstash-plugins/logstash-input-kinesis/pull/29

[jordansissel]

PRs merged. Waiting on a travis cycle so I can publish via Jarvis.

[jordansissel]

published https://rubygems.org/gems/logstash-output-xmpp/versions/3.0.5

[jordansissel]

published https://rubygems.org/gems/logstash-input-gelf/versions/3.0.5

published https://rubygems.org/gems/logstash-input-xmpp/versions/3.1.4

[jordansissel]

published https://rubygems.org/gems/logstash-input-kinesis/versions/2.0.4-java

[jordansissel]

Getting SSL errors and other weird things with the docs plugins list generator.. Sigh.

Doing this manually, and probably inaccurately.

% curl -s https://raw.githubusercontent.com/logstash-plugins/logstash-output-xmpp/v3.0.5/docs/index.asciidoc | sed -e 's/%VERSION%/3.0.5/; s/%RELEASE_DATE%/2017-06-27/; s,%CHANGELOG_URL%,https://raw.githubusercontent.com/logstash-plugins/logstash-output-xmpp/v3.0.5/CHANGELOG.md,' > docs/plugins/outputs/xmpp.asciidoc
% curl -s https://raw.githubusercontent.com/logstash-plugins/logstash-input-gelf/v3.0.5/docs/index.asciidoc | sed -e 's/%VERSION%/3.0.5/; s/%RELEASE_DATE%/2017-06-27/; s,%CHANGELOG_URL%,https://raw.githubusercontent.com/logstash-plugins/logstash-input-gelf/v3.0.5/CHANGELOG.md,' > docs/plugins/inputs/gelf.asciidoc
% curl -s https://raw.githubusercontent.com/logstash-plugins/logstash-input-xmpp/v3.1.4/docs/index.asciidoc | sed -e 's/%VERSION%/3.1.4/; s/%RELEASE_DATE%/2017-06-27/; s,%CHANGELOG_URL%,https://raw.githubusercontent.com/logstash-plugins/logstash-input-xmpp/v3.1.4/CHANGELOG.md,' > docs/plugins/inputs/xmpp.asciidoc
% curl -s https://raw.githubusercontent.com/logstash-plugins/logstash-input-kinesis/v2.0.4/docs/index.asciidoc | sed -e 's/%VERSION%/2.0.4/; s/%RELEASE_DATE%/2017-06-27/; s,%CHANGELOG_URL%,https://raw.githubusercontent.com/logstash-plugins/logstash-input-kinesis/v2.0.4/CHANGELOG.md,' > docs/plugins/inputs/kinesis.asciidoc
% curl -s https://raw.githubusercontent.com/logstash-plugins/logstash-output-websocket/v3.0.2/docs/index.asciidoc | sed -e 's/%VERSION%/3.0.2/; s/%RELEASE_DATE%/2017-06-27/; s,%CHANGELOG_URL%,https://raw.githubusercontent.com/logstash-plugins/logstash-output-websocket/v3.0.2/CHANGELOG.md,' > docs/plugins/outputs/websocket.asciidoc

[jsvd]

After checkin in with @dedemorton, this doesn't seem to be a blocker anymore so I removed the tag, and moved the version tag to 5.6.0

[dedemorton]

@jordansissel Closing this because it's quite old. AFAIK, there are no remaining tasks for this old release.