Screwdriver: Blocked by block job itself

[yuichi10]

What happened:
Under in a situation, build does not start.

What you expected to happen:
Every build starts as usual.

How to reproduce it:

1. start builds several times continuously.
2. stop queued build (For example 3 of the picture)
3. then stop 1 and 2
4. then 4 is not start

[jithine]

@yuichi10 This is a bug with blockedBy self job feature. We have now put this behind a feature flag. Set this https://github.com/screwdriver-cd/queue-worker/blob/master/config/custom-environment-variables.yaml#L148 to "false" in your environment to unblock for now, while we fix the underlying issue.

[d2lam]

@yuichi10 Thanks for making an issue.
We made a couple fixes to mitigate it. Please pull in queue-worker version queue-worker:v1.12.9. and set PLUGIN_BLOCKEDBY_BLOCKED_BY_SELF to "false"

The root cause is here: https://github.com/screwdriver-cd/executor-queue/blob/master/index.js#L307-L323
Previously, there are 2 scenarios: build is either in the queue or picked up by the worker.
If in the queue, we delete it from the queue
If in the worker, we call job stop: https://github.com/screwdriver-cd/queue-worker/blob/master/lib/jobs.js#L75-L96

However, we introduced reEnqueue. The problem happens when a build is aborted and it is being picked up by the worker. Therefore it's not deleted from the queue and gets picked up again. We need to implement extra steps to clean up inside beforePerform.

[yuichi10]

Thanks for your advice!
Now the Screwdriver work well.

[d2lam]

Summary 6/12/2018

Previously, we addressed this case by adding delete key:

• Build is still in the queue and picked up by worker
• User aborts
• Executor-queue adds delete key, and call stop which cleans up waiting key
• Worker puts back to queue
• Worker picks up same build again, in beforePerform, it does clean up and will not proceed.

There is a new edge case scenario I found:

• Build is still in the queue and not picked up by worker
• User aborts
• Executor-queue removes it from queue, adds the delete key, waiting key remains
• Because it's removed from queue, it will not go to beforePerform to clean up. waiting key still contains this buildId, so subsequent builds cannot be started.

Will open PRs to address this new edge case

[d2lam]

@yuichi10 I think this issue is resolved
Please pull in v1.12.11 for worker and set PLUGIN_BLOCKEDBY_BLOCKED_BY_SELF to "true"
Also, please pull in v0.5.370 for the API

[d2lam]

@yuichi10 When possible, can you confirm it is working for you? If it is working, you can close the issue. If it's not working, please let me know! Thank you!

[jithine]

Closing this. Please re-open if issue is seen again.

[yuichi10]

It work well!!
Thank you!!!