Luigi: Worker thinks a failed job is still pending and never quits when keep-alive is enabled

[yuribak]

Luigi version 2.4.0
keep-alive is enabled

I am running into an issue as part of a more complicated workflow, but I have been able to recreate it with a minimal reproduction:

I am submitting a single task with no dependencies that just raises an exception and fails immediately. According to the logs and the luigi scheduler frontend, the failure is identified correctly. However, the worker process still seems to think it has a pending task and just stays alive forever (logs below).

I would have expected the worker's view of the task's status to mirror that of the luigi scheduler's: that there is only one task, it has failed, and there is no reason for the worker to stay up.

This is causing a problem, since we are monitoring the worker's status in order to determine the overall pass/fail of the top-level process in our actual workflow. Can you provide any guidance on what we are doing wrong?

[28] 2017-03-23 20:31:06,967 - luigi-interface - DEBUG - Checking if Fails(domain=mydomain.com) is complete
[28] 2017-03-23 20:31:06,995 - luigi-interface - INFO - Informed scheduler that task   Fails_2016_05_01_per_mgr_s3___devel_sandb_ad1bda6a6c   has status   PENDING
[28] 2017-03-23 20:31:06,996 - luigi-interface - INFO - Done scheduling tasks
[28] 2017-03-23 20:31:06,996 - luigi-interface - INFO - Running Worker with 1 processes
[28] 2017-03-23 20:31:06,998 - luigi-interface - DEBUG - Asking scheduler for work...
[28] 2017-03-23 20:31:07,024 - luigi-interface - DEBUG - Pending tasks: 1
[28] 2017-03-23 20:31:07,025 - luigi-interface - INFO - [pid 28] Worker Worker(salt=811269085, workers=1, host=baf050053ca8, username=root, pid=28) running   Fails(domain=mydomain.com)
[28] 2017-03-23 20:31:07,025 - sciluigi-interface - INFO - Task Fails started
[28] 2017-03-23 20:31:07,025 - luigi-interface - ERROR - [pid 28] Worker Worker(salt=811269085, workers=1, host=baf050053ca8, username=root, pid=28) failed    Fails(domain=mydomain.com)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/luigi/worker.py", line 192, in run
    new_deps = self._run_get_new_deps()
  File "/usr/local/lib/python2.7/dist-packages/luigi/worker.py", line 130, in _run_get_new_deps
    task_gen = self.task.run()
  File "magneton/plumbing/test/test_task.py", line 50, in run
    raise Exception("fail fail fail fail fail fail fail fail fail fail fail")
Exception: fail fail fail fail fail fail fail fail fail fail fail
[28] 2017-03-23 20:31:07,028 - luigi-interface - DEBUG - 1 running tasks, waiting for next task to finish
[28] 2017-03-23 20:31:07,050 - luigi-interface - INFO - Informed scheduler that task   Fails_2016_05_01_per_mgr_s3___devel_sandb_ad1bda6a6c   has status   FAILED
[28] 2017-03-23 20:31:07,050 - luigi-interface - DEBUG - Asking scheduler for work...
[28] 2017-03-23 20:31:07,053 - luigi-interface - DEBUG - Done
[28] 2017-03-23 20:31:07,053 - luigi-interface - DEBUG - There are no more tasks to run at this time
[28] 2017-03-23 20:31:07,053 - luigi-interface - DEBUG - There are 1 pending tasks possibly being run by other workers
[28] 2017-03-23 20:31:07,053 - luigi-interface - DEBUG - There are 1 pending tasks unique to this worker
[28] 2017-03-23 20:31:07,053 - luigi-interface - DEBUG - There are 1 pending tasks last scheduled by this worker
[28] 2017-03-23 20:31:07,053 - luigi-interface - DEBUG - Sleeping for 14.176448 seconds
[28] 2017-03-23 20:31:21,235 - luigi-interface - DEBUG - Asking scheduler for work...
[28] 2017-03-23 20:31:21,238 - luigi-interface - DEBUG - Done
[28] 2017-03-23 20:31:21,238 - luigi-interface - DEBUG - There are no more tasks to run at this time
[28] 2017-03-23 20:31:21,238 - luigi-interface - DEBUG - There are 1 pending tasks possibly being run by other workers
[28] 2017-03-23 20:31:21,238 - luigi-interface - DEBUG - There are 1 pending tasks unique to this worker
[28] 2017-03-23 20:31:21,238 - luigi-interface - DEBUG - There are 1 pending tasks last scheduled by this worker
[28] 2017-03-23 20:31:21,239 - luigi-interface - DEBUG - Sleeping for 13.840423 seconds
[28] 2017-03-23 20:31:35,085 - luigi-interface - DEBUG - Asking scheduler for work...
[28] 2017-03-23 20:31:35,089 - luigi-interface - DEBUG - Done
[28] 2017-03-23 20:31:35,089 - luigi-interface - DEBUG - There are no more tasks to run at this time
[28] 2017-03-23 20:31:35,089 - luigi-interface - DEBUG - There are 1 pending tasks possibly being run by other workers
[28] 2017-03-23 20:31:35,089 - luigi-interface - DEBUG - There are 1 pending tasks unique to this worker
[28] 2017-03-23 20:31:35,089 - luigi-interface - DEBUG - There are 1 pending tasks last scheduled by this worker
[28] 2017-03-23 20:31:35,089 - luigi-interface - DEBUG - Sleeping for 13.317368 seconds
...

[srikiraju]

Worse, if you generate the output/target externally, luigi doesn't attempt to check if a failed task was fixed/overriden externally and keeps retrying

[Tarrasch]

This is expected and for me, usually a good behaviour. Some things to keep in mind:

• In some sense a failed task is also considered pending. It will be retried once (default 15 minute) cooldown is over.
• Luigi also has the concept of disabled task. It's not on by default as it complicates stuff for the first-time users of luigi, but you should consider using this, because disabled tasks are not considered pending, and you worker will happily quit as it believes there's nothing to wait for.

My suggestion would be to consider if configuring disabling of tasks could work for you. Let me know it it works out.

---

@srikiraju -- I think the worker should recheck if the task is complete() or not, if it doesn't maybe it's a bug? :)

[yuribak]

@Tarrasch

• The retry and cooldown settings are known at runtime, i'd expect a task to be marked as failed once the retry count is exhausted.
• I also find the inconsistency between the worker's view of things and that of the scheduler's surprising. If a failed task should still be considered pending then the monitoring UI should reflect that as well.
• I will look into disabling tasks as a workaround.

Thanks for the reply.

[srikiraju]

@Tarrasch It should definitely recheck if the task is complete() or not, but it doesn't.

[Tarrasch]

I also find the inconsistency between the worker's view of things and that of the scheduler's surprising. If a failed task should still be considered pending then the monitoring UI should reflect that as well.

Agreed. Things could be more clear. But please remember that it in general it makes sense for Luigi to retry a failed task. That's why people choose Luigi over just directly calling their external program.

The retry and cooldown settings are known at runtime, i'd expect a task to be marked as failed once the retry count is exhausted.

Wait, retry-count is how many times you run before you disable it. Maybe the lifetime of a Luigi task is different from what you believe it to be?

---

@srikiraju - Please open a new issue, or even better, a PR with failing test case and/or fix.

[veiko99]

Luigi also has the concept of disabled task. It's not on by default as it complicates stuff for the first-time users of luigi, but you should consider using this, because disabled tasks are not considered pending, and you worker will happily quit as it believes there's nothing to wait for.

I have configured disabling tasks. If the run method raises an exception, task gets disabled and worker quits. But if the output method raises an exception, task gets status 'unknown' and worker never quits when keep-alive is enabled.

[jerry-rubikloud]

I just encountered this issue. The unknown task should have failed but somehow the worker thinks it is a pending task. The worker just keeps sleeping as no pending tasks can be executed for over 1 hour. Any fix for this would be greatly appreciated.

[alexeygrigorev]

We also have encountered exactly that: instead of failing the task and exiting the flow, worker keeps waiting for something to happen.

The tasks with usual failures are handled fine and marked as "disabled". This allows to end the flow with a failure. This works ok.

But sometimes there is another kind of failure, e.g. with configuration. Then the task's status becomes "unknown", the upstream tasks are "pending" and nothing is happening. So we have to go to the machine and manually restart the worker. One particular example when it happens is when we forget a trailing slash in S3FlagTarget.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.

[afaq404alam]

What was the fix for this issue? Currently, I am also facing a similar issue. When a task fails in requires() method, it goes into UNKNOWN state and with keep-alive, luigi just waits forever without retrying for this task.