Prefect: set task dependency

Created on 30 Mar 2020  路  3Comments  路  Source: PrefectHQ/prefect

Current behavior

I would like to do simple for loop with prefect,

with Flow('task') as flow:
    dt_list = get_date()
    dt = task1.map(dt_list)
    task2.map(dt_list)

but get this dependence:

Task 'task1[0]
Task 'task1[1]
Task 'task1[2]
....
Task 'task2[0]'
Task 'task2[1]'
Task 'task2[2]'

try to use set_upstream or trigger any_suceesful but did't work too

with Flow('task') as flow:
    dt_list = get_target_date()
    dt = task1.map(dt_list)
    task2.map(dt_list)
    task2.set_upstream(task1)

Proposed behavior

How Can I bind the task dependence between task1 and task2 like this?
Thanks

Task 'task1[0]'
Task 'task2[0]'
....
Task 'task1[1]'
Task 'task2[1]'
....
Task 'task1[2]'
Task 'task2[2]'
....
...
enhancement

Most helpful comment

@ponggung I believe what you are looking for is what we call "Depth First Execution": #2041

We plan to prioritize this work for one of the next releases!

All 3 comments

@ponggung I'm not sure I fully understand what you're trying to do here but Prefect does have a concept of looping with dynamic DAGs and this may help achieve your goal

@ponggung I believe what you are looking for is what we call "Depth First Execution": #2041

We plan to prioritize this work for one of the next releases!

@joshmeek @cicdw
Yes, Depth First Execution is exactly what I am looking for !
Thank you for your kindly reply and information,
and I expect the next releases to come.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

emcake picture emcake  路  3Comments

dkapitan picture dkapitan  路  3Comments

mark-w-325 picture mark-w-325  路  3Comments

GZangl picture GZangl  路  3Comments

jlowin picture jlowin  路  4Comments