Luigi: Luigi triggers run method twice for tasks yields some dynamic dependencies

[javrasya]

Whenever a task yields a dynamic dependency, it's run method triggered twice. Here is a case to produce it;

from unittest import TestCase

from luigi.scheduler import Scheduler
from luigi.worker import Worker

from tests.helpers import DummyTask

_author_ = 'ahmetdal'


class TriggeringRunTwiceCase(TestCase):
    def setUp(self):
        self.sch = Scheduler()

    def test_with_dynamic_dependencies(self):
        outputs = []

        class B(DummyTask):
            def run(self):
                super(B, self).run()
                outputs.append('B Run')

        b = B()

        class A(DummyTask):
            def run(self):
                super(A, self).run()
                outputs.append('Before yield B')
                yield b
                outputs.append('After yield B')

        a = A()

        with Worker(scheduler=self.sch) as w:
            self.assertTrue(w.add(a))
            self.assertTrue(w.run())

            expected_outputs = [
                "Before yield B",
                "B Run",
                "After yield B",
            ]

            self.assertListEqual(expected_outputs, outputs)

Output of this test;

Second list contains 1 additional elements.
First extra element 3:
'After yield B'

- ['Before yield B', 'B Run', 'After yield B']
+ ['Before yield B', 'B Run', 'Before yield B', 'After yield B']
?                            ++++++++++++++++++

As you can see, the part before the yield is running twice (not the part after the yield.). 'Before yield B' output seems more than one.

[erikbern]

This is expected behavior. The reason is that execution state is not possible to transfer between processes. If a new worker or process wants to resume there's no way. So they just restart from scratch. That's why tasks need to be idempotent for it to work.

We could potentially optimize it so that there's no reason to restart in case it's the same worker that resumes it, but that's a heuristic optimization.