Conductor: BUG - MySQLExecutionDAO.addEventExecution wrong index

Created on 23 Feb 2019 · 4Comments · Source: Netflix/conductor

Problem: cannot use more than one action in the same EventHandler with MySQL persistence

Reason: UNIQUE KEY unique_event_execution (event_handler_name,event_name,message_id)

Detailed description: when a new incoming message that triggers a new workflow execution is processed by EventProcessor, it tries to extract the list of actions defined inside the EventHandler definition and execute them asyncronously. But the problem is before executing every action, it tries to add a new record to the "event_execution" table in MySQL. The unique index used in that table is the same for both actions because it doesn't use execution_id but uses just a message_id instead, so it cannot add more than one execution for this message_id and throws exception.

Throwing exception leads to even more serious problem: the incoming event will never be acknowledged, even though the first action was successfully added to the table and was successfully executed, but the acknowledment for the whole event happens only when there is no exception for both actions.

Also, when next time (after unack timeout) the same event will come again, no actions can be executed and nothing can be changed because the record is already in the table so this time it will be an exception for the first action immediately. It leads to messages constantly hanging in the input queue.

Expected behavoir: the unique index for this MySQL table should be per action, not per message_id. It should work properly if to use "execution_id" instead of "message_id" inside the index, because "execution_id" contains a suffix that is incremented for every action inside the event handler.

Also, if somehow finally we got the exception, after unack timeout when we get the same incoming event again, we should be able to either acknowledge it or execute it again, or it will hang in the incoming queue forever as it happens now.

Example of EventHandler definition to reproduce the problem:

{
  "name": "trigger_test_workflow",
  "event": "sqs:my-event",
  "condition": "true",
  "active": true,
  "actions": [
    {
      "action": "start_workflow",
      "start_workflow": {
        "name": "my-workflow",
        "version": 2,
        "input": {
          "receipt_id": "${receipt_id}",
          "sent_at": "${sent_at}"
        }
      }
    },
    {
      "action": "start_workflow",
      "start_workflow": {
        "name": "my-workflow",
        "version": 1,
        "input": {
          "receipt_id": "${receipt_id}",
          "sent_at": "${sent_at}"
        }
      }
    }
  ]
}

bug

Source

dskarataev

All 4 comments

@picaron maybe you could have a look and tell your thoughts, thanks

dskarataev on 23 Feb 2019

@dskarataev
This seems like a reasonable approach, to have the unique be on the execution_id instead of message_id. If you want, you can submit the PR for it, or else I can do it.