Argo: Mutex not being released on step completion

[davidcollom]

Summary

When using a variable within the mutex name, Mutex's aren't released until workflow-controller is restarted.
Within the example below, the workflow never completes and waits pending on the final unlock.

Additionally all other workflows aren't able to continue from the initial mutex locked (from gen-number-list).

Restarting the workflow-controller brings the workflow to life and all locks are released.

Diagnostics

GKE - v1.17.12-gke.1504

Argo version: v2.12.2

Workflow: https://gist.github.com/davidcollom/9c6f7d9d1819fe922d21b3a69e561754#file-workflow-yaml

Logs: https://gist.github.com/davidcollom/9c6f7d9d1819fe922d21b3a69e561754#file-logs

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[simster7]

@sarabala1979 ?

[sarabala1979]

I will look

[davidcollom]

@sarabala1979 Something I didn't highlight in my initial report was that one/some of the steps are daemon'd steps, I have a feeling that this could be a good line of investigation of this issue. (I haven't confirmed this)

[simster7]

Possibly related to https://github.com/argoproj/argo/pull/4835?

[sarabala1979]

Let me verify that. I am able to reproduce locally. I am investigating