Task: Multiple tasks with the same dependency should run dependency once!

[smyrman]

UPDATE: This is a general feature request to not run _equal_ dependencies more than once. E.g. if task A and task B both depend on taskC, running _both_ task A and task B in parallel (witin one instance of task), should be able to _resolve_ the dependencies so that taskC is only run once, and this is the interesting port, _if it's equal_, which we will get to next.

what makes a task equal?

Since you can _pass variables_ to tasks, it's not enough to just look at the name. The _best_ option is probably to rely on a hash of the name + _sorted_ and _resolved_ input variables. A close second could be to rely on _unresolved_ but still _sorted_ input variables, which is not perfect, but could be an acceptable trade-off if it's somehow easier in code.

---

Original description

I have multiple tasks in my Taskfile that depend on a single task called go-dep that runs the command glide install. When I run task to build multiple targets, I would expect go-dep to be called at most 1 time (zero if it's found to be up-to-date). Instead, I see it's called once for each task that depend on it:

$ task
task: No argument given, trying default task
glide install
glide install
glide install
glide install

PS! go-deps in this case is called with no vars, but the rule should really apply to run a dep at most 1 time when the vars are _exactly_ the same.

[andreynering]

Hmm... I'm not sure I agree with this. Some users may want to re-run a dependency each time (for example, if another task changed any file and something has to be re-generated).

Wouldn't this work for you?

go-dep:
  cmds:
    - glide install
  sources:
    - glide.yaml
    - glide.lock
  generates:
    - vendor/**/*

I'll keep this open to see if anyone have an opinion on this.

[smyrman]

Thats what I have, but when it _is_ out og date, and multiple tasks run as deps (say 4) call it simultaniously, it's launcher 4 times concurrently, which isn't just unececary work, it does not work as _git_ is not built for it. I guess I could work around it, but it's a bit iffy:

default:
  deps:
    - go-deps # ensure up-to-date before launching other tasks
  cmds:
    - task: actual-default
actual-default:
  deps:
    - cmd/go-binary1 # these actually depends on go-deps
    - cmd/go-binary2
    - cmd/go-binary4
    - cmd/go-binary3

If a dependent can not be resolved to one call inn this case, it's not really a dependent at all, just a concurrent-task.

[andreynering]

Maybe we get a solution to this someday. I'm don't have anything in mind now.

By now, you can workaround it in at least two ways:

# remove `go-deps` as a dependency of each build
build-all:
  deps:
    - go-deps
    - cmd/go-binary1
    - cmd/go-binary2
    - cmd/go-binary4
    - cmd/go-binary3

# run serially instead of concurrently
# e.g.: use `cmds` instead of `deps`
build-all:
  cmds:
    - task: cmd/go-binary1
    - task: cmd/go-binary2
    - task: cmd/go-binary4
    - task: cmd/go-binary3

[smyrman]

Your first suggestion does not work, because go-deps will be started at the same time as all cmd/go-binaryN tasks, and thus the dependencies may not be up-to-date when the cmds compile.

The second solution works, but makes for a less efficient pipeline, and I would not be able to run a subset of tasks taks cmd/go-binary1 cmd/go-binary2.

[smyrman]

To be clear, dependency resolution, in my mind at least, should resolve to one task only if the variables are _exactly_ the same, and only within the application's run-time. Thus, this example should also work:

default:
  deps:
    - task: go-compile
      vars: {DIR: cmd/go-binary-1}
    - task: go-compile
      vars: {DIR: cmd/go-binary-2}   # run again as the variables are different.
    - task: go-compile
      vars: {DIR: cmd/go-binary-3}   # same here
go-compile:
  deps:
    - go-deps   # resolve and run once
  dir:  "{{.DIR}}"
  cmds:
    - go install .
go-deps:
  cmds:
    - glide install

@andreynering , it's ok that you don't have a solution for it right now, but it would be nice to know if you agreed if this would be the best way for this to work. There will always be a way to implement it.

I suggest we can have it in the back of our mind when looking into #45. One way to do it would be to have a mutex stored on a unique hash taken from the variables and task-name in the executor, and ensure that a task with a given set of variables, when run as a dependency only, is not run concurrently. The mutex would need to _block_ the task's is-up-to-date check for all other tasks, so that they would _wait_ for the dependency to complete rather than re-launch it.

[andreynering]

Your first suggestion does not work, because go-deps will be started at the same time as all cmd/go-binaryN tasks, and thus the dependencies may not be up-to-date when the cmds compile.

Sorry, I meant:

build-all:
  deps:
    - go-deps
  cmds:
    - task: really-build-all

really-build-all:
  deps:
    - cmd/go-binary1
    - cmd/go-binary2
    - cmd/go-binary4
    - cmd/go-binary3

I know this doesn't solve all points you listed above.

[smyrman]

FYI: updated the description.

[AkihiroSuda]

Any workaround?

[smyrman]

The best workaround for now may be to rely on a _lock_ file for the task that is depended on multiple times in parallel, in addition to some sort of cache to check if the task is done once the lock is required.

See e.g.: https://stackoverflow.com/questions/185451/quick-and-dirty-way-to-ensure-only-one-instance-of-a-shell-script-is-running-at

Some programs, such as dep for Golang dependencies, already uses a lock-file so that no special work-around is needed.

Other then that, it's writing a PR to resolve the issue:-) To retain backwards compatibility you might want to add a parameter to turn the behaviour explicitly on for task version 2.X, and perhaps make it a default for version 3:

version: "2"
deps_resolution: true
...

[AkihiroSuda]

Thanks, but I think the parameter name should not be dep-specific.

Maybe it should be like only_once?

e.g.

version: '2'

tasks:
  myinit:
    only_once: true
    cmds:
      - mount /foo /bar
  myjob1:
    deps: [myinit]
    cmds:
      - mycmd1  
  myjob2:
    deps: [myinit]
    cmds:
      - mycmd2 

[smyrman]

That could makes sense @AkihiroSuda. We might want to extend it a bit to work well with more use-cases though.

One thing to consider is what do you interpret as only once? Is it run the _named task_ that should be run once, or the _named task with a given set of parameters_....

Lets say we change your example slightly:

version: '2'

tasks:
  myinit:
    only_once: true
    cmds:
      - mount '{{.FOO}}' '/bar/{{.FOO}}'
  myjob1:
    deps:
      - task: myinit
        vars: {FOO: "foo"}
    cmds:
      - mycmd1  
  myjob2:
    deps:
      - task: myinit
        vars: {FOO: "bar"}
    cmds:
      - mycmd2
  myjob3:
    deps:
      - task: myinit
        vars: {FOO: "foo"}
    cmds:
      - mycmd3 

How many times should myinit run?

What if the input parameter is considered _insignificant_, such as verbose on/off for a command?

version: "2"
tasks:
  myinit:
    only_once: true
    cmds:
      - mount{{if .VERBOSE}}-v {{end}} '{{.FOO}}' '/bar/{{.FOO}}'

Maybe we need to tell myinit which variables to consider when resolving only once. Something like this:

version: "2"
tasks:
  myinit:
    only_once: true
    only_once_vars:
     - FOO
    cmds:
      - mount{{if .VERBOSE}}-v {{end}} '{{.FOO}}' '/bar/{{.FOO}}'

OR maybe it's good enough to just assume _all paramters_.

PS! ☝️ Syntax and naming could improve, this is just an example.

[smyrman]

Regarding only_once_vars, these are actually also useful for task up-to-date calculation, and naming should reflect this. task_id?

Anyway it could also be an option to just add support for only_once now without considering call parameters (it would already add value), and consider something ala only_once_vars to be contributed in a separate PR and tracked by a s separate issue.

[AkihiroSuda]

+1 for only_once_vars, maybe as task_id_hash_sources?

[smyrman]

@andreynering, any views?

[andreynering]

An UX note: I prefer short keywords where possible, so only_once_vars or task_id_hash_sources are names I think it's ugly

What do you guys think about automatically hashing a task? We could hash all properties that accepts interpolation like cmds, sources, generates, etc. If output is different, task should run again

Maybe that should be configurable like this: a run property with options: once (really once), when_changed (different hashes from previous runs) or always (the default)

(The above is an example, we can change these names to something different, if better)

[smyrman]

My initial reaction is very positive, but there are som subtleness it coudl be nice to clear up.

property with options: once (really once)

Really once within one _run_ of the task executable I presume.

when_changed (different hashes from previous runs)

Also within one _run_ of the task executable, or does it span multiple runs? Also, options or flags that does not affect the output, would trigger a rerun (e.g. verbose on/off) - is that a problem, or an acceptable trade-off?

I _like_ the hashing of tasks, so I think it's worth the trade-off, but there are some subtleness of the semantics it would be good to clear up -- Note that except for a little-bit annoying output, if up-to-date calculation for a task _works_ (has sources, generates set etc.), all that's really needed is that two tasks with the _same hash_ don't run concurrently at run-time, as all other runs of the exact same task with the exact same parameters would be considered up-to-date. However, it would be very _practical_, if it would also _work_ when the task _lacks_ up-to-date calculation parameters, and then it's the case that if a task with the same hash has already been started, don't start another instance (if run == when_changed). Note that this is all _runtime_.

I think we need to separate the concerns of per-run deduplication and up-to-date calculation sligthly, although there will still be some overlap.

Runtime de-duplicatoin

So let's advance on that. although deduplicate is a long name, I picture what we want in terms of _semantics_ is perhaps closer to this:

• deduplicate: "no" (default)
• deduplicate: "hash"
• deduplicate: "name"

Implementation-wise, I picture something like this:

• A map[string]context.Context is used to guard/track running tasks with deduplicate set to "hash" or "name".
• When a task with deduplicate starts, it checks (in a thread-safe way) if there is already a context in the map.
  
  
  
  • If no, create a new context
    
    ctx, cancel := context.WithCancel(ctx) that is cancelled when the task completes, and run the task.
  
  
  • If yes, wait for the previous context to be cancelled (complete), and return the error, if any.
  
  

Or something along those lines.

Up-to-date calculation

For _caching task status between runs_, the hash of the task should be used as a key over the task-name when
method: checksum is used. I.e. replace .task/checksum/<task name> -> .task/checksum/<task hash>.

If backwards compatibility is _needed_ for the up-to-date calculation, we could to this only when method: checksum2, but personally I don't think it is at this level.

The benefit of this over today's solution, is that it allows for multiple calls to the same task could all be cached and considered up to date. E.g.
```yaml:

• task: myinit
  vars: {FOO: "foo"}
• task: myinit
  vars: {FOO: "bar"}
  ```

[Evertras]

Is there still interest in tackling this? I'm enjoying using Taskfiles so far, but I'd really like to be able to safely declare dependencies without having to worry about how they chain and still take advantage of running things in parallel.

I would also like to argue against 'run always' as the default. I can't think of any situation where a task being run with the same variables would need to run multiple times in parallel. I know that would be a breaking change for v2, but I'd strongly argue for it being 'run only once' for v3 with the option to let it run multiple times still available.

[Evertras]

As another thought, would it simplify things to simply guard against tasks with duplicate 'generates' outputs running in parallel?

[danielecook]

I am still very interested in this feature.

[mknapik]

@smyrman mentioned using lock files. Not a perfect solution but a nice workaround for the time being:

tasks:
# ...
  compose:build:
    cmds:
      - flock -w 600 task.compose:build.lock docker-compose build --parallel
# ...

flock should be available out of the box in most Linux distros. On macOS it's possible to install it via brew.