Nomad: Improvement: Nomad should be able to schedule tasks in order

[sebamontini]

It should be awesome to be able to schedule the tasks in order as they are defined, or at least enable some sort of configuration in a group tasks to ensure dependency between tasks, something like "when this tasks finishes, trigger this other tasks" or at least be able to do so with jobs

[dadgar]

Are you asking to do that within a task group or between task groups/jobs?

[sebamontini]

within a task group would be great I think. So you can create a job with a task group like

job process-data:
  group1:
    task1: download stuff
    task2: process that stuff
    tasks3: upload resulting data to another place

and have them run in order and one as a dependency of another.

but being able to do so between jobs would be ok also (if it's easier to be implemented that way), to be able to trigger a job2 after job1 has finished succesfully (and have one job per task)

[afterdesign]

This is exactly the usecase I wanted to use nomad for. As I've in debug log every task is spawning new process so this might be a bit problematic to do. But if there could be option to create process only for one task at a time then this could work.

[shantanugadgil]

@dadgar

I think, 'chaining tasks within a job' and creating dependencies among jobs, both would be useful features.
Something analogous to Terraform's "depend_on" 😄
(task 2 runs only after task 1 finishes, etc.)

[dreh23]

This looks like a duplicate of #1061 (and #2767) or at least it's related.

[sebamontini]

@dreh23 it's not exactly the same, those are hooks or commands to ensure proper setup of the environment (or cleanup) for some task, here I'm asking to be able to chain tasks or even jobs (similar to Airflow) to be able to set precedence or even dependencies between jobs/task to build a whole procesing pipeline in some cases

[dadgar]

This is a duplicate of https://github.com/hashicorp/nomad/issues/419

[sagarrakshe]

An attempt to address this issue: https://github.com/sagarrakshe/nomad-dtree