Logstash: 6.4.0 pipeline reloading breaks with PQ enabled

Created on 11 Sep 2018  路  5Comments  路  Source: elastic/logstash

This is a regression from 6.3.2.

IF PQ is enabled and a pipeline reload is triggered (using config.reload.automatic: true) it results in this exception

[2018-09-11T10:22:51,176][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /Users/colin/dev/src/elasticsearch/logstash-6.4.0/data/queue/main/.lock
    at org.logstash.ackedqueue.Queue.open(Queue.java:262) ~[logstash-core.jar:?]
    at org.logstash.ackedqueue.ext.JRubyAckedQueueExt.open(JRubyAckedQueueExt.java:97) ~[logstash-core.jar:?]
    at org.logstash.ackedqueue.ext.JRubyWrappedAckedQueueExt.initialize(JRubyWrappedAckedQueueExt.java:39) ~[logstash-core.jar:?]
    at org.logstash.ackedqueue.QueueFactoryExt.create(QueueFactoryExt.java:37) ~[logstash-core.jar:?]
    at org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:130) [logstash-core.jar:?]
    at org.logstash.execution.AbstractPipelineExt$INVOKER$i$3$0$initialize.call(AbstractPipelineExt$INVOKER$i$3$0$initialize.gen) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodThree.call(JavaMethod.java:1186) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:743) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:983) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:974) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39) [jruby-complete-9.1.13.0.jar:?]
    at Users.colin.dev.src.elasticsearch.logstash_minus_6_dot_4_dot_0.logstash_minus_core.lib.logstash.pipeline.RUBY$method$initialize$0(/Users/colin/dev/src/elasticsearch/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:22) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.RubyClass.newInstance(RubyClass.java:1022) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145) [jruby-complete-9.1.13.0.jar:?]
    at Users.colin.dev.src.elasticsearch.logstash_minus_6_dot_4_dot_0.logstash_minus_core.lib.logstash.pipeline_action.reload.RUBY$method$execute$0(/Users/colin/dev/src/elasticsearch/logstash-6.4.0/logstash-core/lib/logstash/pipeline_action/reload.rb:35) [jruby-complete-9.1.13.0.jar:?]
    at Users.colin.dev.src.elasticsearch.logstash_minus_6_dot_4_dot_0.logstash_minus_core.lib.logstash.pipeline_action.reload.RUBY$method$execute$0$__VARARGS__(/Users/colin/dev/src/elasticsearch/logstash-6.4.0/logstash-core/lib/logstash/pipeline_action/reload.rb) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145) [jruby-complete-9.1.13.0.jar:?]
    at Users.colin.dev.src.elasticsearch.logstash_minus_6_dot_4_dot_0.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/Users/colin/dev/src/elasticsearch/logstash-6.4.0/logstash-core/lib/logstash/agent.rb:309) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.runtime.Block.call(Block.java:124) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.RubyProc.call(RubyProc.java:289) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.RubyProc.call(RubyProc.java:246) [jruby-complete-9.1.13.0.jar:?]
    at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104) [jruby-complete-9.1.13.0.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
[2018-09-11T10:22:51,185][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /Users/colin/dev/src/elasticsearch/logstash-6.4.0/data/queue/main/.lock", :backtrace=>["org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:133)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$3$0$initialize.call(AbstractPipelineExt$INVOKER$i$3$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodThree.call(JavaMethod.java:1186)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:743)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:983)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:974)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "Users.colin.dev.src.elasticsearch.logstash_minus_6_dot_4_dot_0.logstash_minus_core.lib.logstash.pipeline.RUBY$method$initialize$0(/Users/colin/dev/src/elasticsearch/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:22)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.RubyClass.newInstance(RubyClass.java:1022)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "Users.colin.dev.src.elasticsearch.logstash_minus_6_dot_4_dot_0.logstash_minus_core.lib.logstash.pipeline_action.reload.RUBY$method$execute$0(/Users/colin/dev/src/elasticsearch/logstash-6.4.0/logstash-core/lib/logstash/pipeline_action/reload.rb:35)", "Users.colin.dev.src.elasticsearch.logstash_minus_6_dot_4_dot_0.logstash_minus_core.lib.logstash.pipeline_action.reload.RUBY$method$execute$0$__VARARGS__(/Users/colin/dev/src/elasticsearch/logstash-6.4.0/logstash-core/lib/logstash/pipeline_action/reload.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "Users.colin.dev.src.elasticsearch.logstash_minus_6_dot_4_dot_0.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/Users/colin/dev/src/elasticsearch/logstash-6.4.0/logstash-core/lib/logstash/agent.rb:309)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]}
blocker bug
Source
picture colinsurprenant

All 5 comments

@colinsurprenant since the latest PQ implementation has been backport to the 5.6 branch, can you check if 5.6.11 is also affected by this issue?

jsvd picture jsvd on 11 Sep 2018
👍1

I think I just found the cause. preparing a fix.

colinsurprenant picture colinsurprenant on 12 Sep 2018
❤1

I have a fix working locally. running tests and pushing a PR shortly.

colinsurprenant picture colinsurprenant on 12 Sep 2018

PR #9985 submitted

colinsurprenant picture colinsurprenant on 12 Sep 2018

backports for 6.4 #9986 and 6.x #9987 submitted

colinsurprenant picture colinsurprenant on 13 Sep 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

cschotke picture cschotke  路  3Comments
molitoris picture molitoris  路  3Comments
bertramn picture bertramn  路  3Comments
jsvd picture jsvd  路  3Comments
amodakvnera picture amodakvnera  路  3Comments