Conductor: Workflow sometimes fails to transition to COMPLETED

[xesxen]

Given the attached sample workflow, Conductor sometimes fails to transition to COMPLETED. Even weirder, I've seen the workflow transition from COMPLETED back to RUNNING when refreshing the web interface quickly after starting this workflow.

In the workflow state, the last task (placeholder_false, which is an event) is marked as COMPLETED, but the state of the workflow itself is still RUNNING.

Attached below is a sample workflow, task definition (HTTP) and workflow state. Note: it doesn't trigger every time, but starting multiple instances of the workflow should trigger it.

Used versions:

• Conductor: v1.11.5
• Dynomite: f77f2178da46c54d6b2ee2a48866c17ccecd03b7
• Elasticsearch: 5.6.11

workflow-dummy.json.txt
workflow-state.json.txt
task_fetch_site.json.txt

[saidatta]

For what its worth, I encountered that too. I dont encounter this for now after I made the changes to dynomite or switched to Elasticache.

[jwulf]

I observe this behaviour when running in-memory - and even more so when running with Dynomite via docker-compose - and starting 100 workflow instances at a time.

Workflows also hang at the start point - they are returned by querying running workflows, but no tasks for them are in the queue.

[jwulf]

I find Workflows with all tasks completed. If I pause the workflow manually in the UI, then resume it, it gets correctly marked as completed.

[cyzhao]

@apanicker-nflx do we still have this problem?

[nullp0int3r]

same issue with docker-compose and Mysql

[jwulf]

This made Conductor unusable for us. We went with Zeebe. They have different priorities from the Conductor team and respond within minutes or hours on GitHub.

[nullp0int3r]

This made Conductor unusable for us. We went with Zeebe. They have different priorities from the Conductor team and respond within minutes or hours on GitHub.

Thank you! I'm gonna have a look ;)

[jmaganATS]

I have the same problem with this workflow:

In addition, all these executions are identical, therefore, they should have the same value in "Failed Tasks" field ("task_2, task_3"), but it's random. It seems that the threads are overwriting the Status and Failed Tasks values.

[dskarataev]

The same problem here after a very small performance test 5 workflows/second for 10 minutes.

[garreananth]

Even I have encountered this problem when all subtasks are COMPLETED the workflow is still showing as RUNNING, is their any workaround for this (other than clicking PAUSE and RESUME) ?

Conductor Version: 1.9.0-SNAPSHOT | Build Date: 2018-05-07_17:26:13

[saidatta]

@dskarataev Can you try binding the workflowSweeper as an eagerSingleton in ServerModule or Try the latest conductor-server that has that fix?

[kishorebanala]

Apologies for delays, We've been running tests to identify root cause of this issue. It is primarily caused by concurrent executions of decide on a workflow. We're working on a fix using distributed locks as interface, and will try to push it as soon as possible. Thanks for your patience.

[rayscott007]

Thanks for taking a look at this issue. Just wondering if there is any update? We are getting about 10% of our workflows finishing all tasks but not completing. We are running the following:

Version: 2.6.0-SNAPSHOT | Build Date: 2019-01-11_23:55:02
DB: Redis

[boshido]

(3 pods in K8S) Conductor Server/UI version: 2.10.0
(AWS Elasticache)Redis version: 3.2.10
(AWS Elasticsearch)Elasticsearch version: 5.6

I faced similarly issue

We tried to test creating workflow around 30000 workflows and let worker completed every workflow immediately.
We faced that around 0 - 300 workflow that stuck at running state even every tasks already done.

This issue can be fixed manually by pause and then resume workflow, workflow itself will change state to complete.

So now, I am still waiting for updates from conductor team about this case.

[boshido]

If this case happens in very low concurrent request.
I'm a bit curious, How your guys handle this case base on one of contributor said "there are around 1,000,000 workflow executed in each day" ?

And any ETA for solving this issue ?, it cause a lot of problem in real world.

[kishorebanala]

Update: We would like to provide a simple implementation that works with existing Dynomite cluster. Dynomite team is looking into providing Redlock implementation in dyno-client, which conductor can piggy back on when available. This is the best possible implementation that could provide reasonable guarantees and works for most of the existing users. ETA for this feature is dependent on Dynomite's implementation.

Meanwhile, we shall provide unimplemented locking interface, which users can implement with their own locking infra, like Redlock + Redisson or similar implementations. Zookeeper might work, provided we can acquire and release lock from same thread. We'll give this a try, and might provide a Zookeeper lock implementation later (Low priority as not everyone might have access to a Zookeeper cluster).

Please understand the risks involved in doing so, and exercise caution when plugging in implementations. ETA for locking interface is first or second week of May.

[boshido]

@kishorebanala Thank you for an update :3
Base on what you said
I'm about to ask just for clarifying

We use Redis instead of Dynomite we have to wait for implementation of locking interface, right ?

But for someone else who use Dynomite, they need to wait for Dynomite team provide RedLook for handle this issue, right ?

[kishorebanala]

@boshido That's correct. If you're already using Redis cluster and have access to any Redlock implementation, you can plug it in to the interface. For folks using Dynomite, we're actively working with Dynomite team to implement Redlock algorithm in dyno-client.

Tagging @milongapau here, who is currently looking into locking interface implementation.

[mudit3774]

@kishorebanala : is there an update in the timelines for this? It was initially first or second week of may. Also, is there a detailed design proposal so that the community can review and add comments ? It will also help us to think of the concrete implementation for say, Redis

[kishorebanala]

@mudit3774 Apologies for slight delays in providing locking interface implementation. @milongapau is working on this feature, and we'll open a PR ASAP. Also, I've opened above #1137 to start a conversation.

Thanks for the suggestion of design proposal. Given this is one of the critical features, we'll publish our notes outlining overall design and edge cases to be solved to #1137.

[mudit3774]

@kishorebanala thanks for considering a fix. We do understand that this is a community project and there could be delays.

Thanks for considering the suggestion for publishing implementation notes.

[rabonja]

@mudit3774 @milongapau wanted to ask if there was any time frame for this fix? I am seeing this happen on version 2.10.1. Seems to happen like 5-10% of the time. All task are completed but the workflow still shows as running.

[milongapau]

@rabonja what do you use for data persistence? Our fix has 2 parts:

1. publish locking API with no-op implementation (will have a PR in couple days)
2. work on Dynomite implementation (depends on Dynomite team's schedule, currently looking at end of quarter)

We take this approach because some in the community will implement their own solution, likely in parallel with our efforts. Depending on your tech stack and timeframe, you can piggyback off one of us or implement your own.

[rabonja]

@milongapau we are using redis.

[smullins3000]

I'm very unclear on what the repro is for this bug. Does anyone have a description of what triggers this behavior? I saw this description of "caused by concurrent executions of decide on a workflow" but I can't quite make sense of that.
The linked issue does contain a better description of the root cause of the bug: https://github.com/Netflix/conductor/issues/1137#issuecomment-493268111

[naveenchlsn]

@milongapau @apanicker-nflx @kishorebanala I see that PR addressing this issue https://github.com/Netflix/conductor/pull/1158/files has not been merged into master. Do we have an expected date when this would be available?

[kishorebanala]

@naveenchlsn We're running regression tests in our test stack with zookeeper, and were fixing bugs as we see them. We're hoping to merge this to master sometime next week.

[rayscott007]

@kishorebanala Will this fix also apply to MySQL backed setups? Or is it only for Dynomite?
Thanks for your work on this

[skyrocknroll]

Does it work for redis cluster also ?

[kishorebanala]

@rayscott007 @skyrocknroll Please see @milongapau's comment for more details about this feature: https://github.com/Netflix/conductor/issues/1137#issuecomment-493268111

To summarize, you'll need a distributed locking provider (eg. Zookeeper) and a database that supports fencing tokens. It is possible to use none, one or both of the locking and fencing tokens to achieve desired levels of workflow consistency.

[kishorebanala]

@smullins3000 So, WorkflowExecutor.decide is the core that drives Workflow execution. It is possible to trigger this decide on a Workflow concurrently. For eg., an updateTask and a WorkflowSweeper can both call decide at exact same time.

Depending on the state a workflow is in, this concurrent decides can cause inconsistencies in execution. One such example is keeping workflow in running state even after all the tasks are completed.

We're trying to solve these inconsistencies by allowing only one decide at a time on a workflow. Given the distributed set-up of Conductor server, this is possible only through a distributed locking service and a database than can support fencing tokens (To avoid node with expired locks to reactive, say after a long JVM pause and continue execution with state data).

[kishorebanala]

Update: We're putting together documentation for locking service, and the decision log. The tests are looking good, and we're planning a beta release ASAP (minus long-weekend distractions).

For users who'd like to play around with the feature, please checkout v2.13.4-alpha.1, which has latest changes to locking service (without fencing tokens) and been recently rebased against master. Ping us in Gitter if you need further help with this.

[saidatta]

As a related aside, Some info on distributing locking in case anyone wanted to catch up or intrigued.

https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html

[kishorebanala]

We've released a beta with locking service implementation using Zookeeper, which solves this issue.

Closing this issue, please continue the discussion in this thread to avoid duplication: https://github.com/Netflix/conductor/issues/1137#issuecomment-532033287