Task: Allow commands run in parallel

[sascha-andres]

For some startup tasks or cleanup paths commands could run in parallel while others should be run in order.
Add a Parallel property to task to allow switching between modes

[sascha-andres]

@andreynering Pushed a first implementation. Obvious point still missing: proper handling of errors. What do you think?

[sascha-andres]

@andreynering Have you tried building with 1.9.2? Due to slimsag/godocmd/testdata ( multiple packages in one dir ) I can not install with task install:

go install -v -ldflags="-w -s -X main.version=c308c52" ./cmd/task
task.go:14:2: found packages a (a0.go) and b (b.go) in /home/andres/go/src/github.com/slimsag/godocmd/testdata
task: Failed to run task "install": exit status 1

I do not have a machine with 1.9.1 at hand to see whether it is really a 1.9.2 change...

[andreynering]

@sascha-andres Hi and welcome back!

About this proposal, I'd prefer to add an cmd: option to dependencies listed in deps:. Dependencies already run in parallel. It would look like this:

task:
  deps:
    - cmd: echo "Hello"
    - cmd: echo "World"

You mentioned cleanup. I also have the idea to have an defer: option on commands. It would always run, on success or error:

task:
  cmds:
    - defer: rm -rf ./tmp
    - mkdir -p ./tmp
    - touch ./tmp/foo.txt

The error you mentioned is because you added that package: https://github.com/go-task/task/commit/c308c52d274368307ee063d08a022fd6ccd02703#diff-8331617cd00da043cb07ad65466dfd15R14

[sascha-andres]

@andreynering Not sure why this package was added...

I take a look at using this with deps, though communication wise I think it's clearer for to have a parallel flag.

The defer idea is great, maybe I take a look at it

[sascha-andres]

@andreynering The idea having a cmd: flag for dependencies is great, as one would not have to wrap a single command dependency in a task of it's own

Regarding my use case:

I plan to use task for pre-push hooks my employers git server. This involves a certain understanding for the IT department, the developers and the guys doing a mixture of operating and development. Therefore I think doing the actual work in dependencies is surprising as it is not what you would expect.

So I think we should support both options.

[andreynering]

@sascha-andres

My worry is having duplicated development effort. For example, I also had the idea of an throttle: option on a task to control how many dependencies run at the same time (useful to limit CPU/memory intensive work). Future customizations like this would have to be added for both deps and parallel cmds.

Anyway, both cmd: on deps and defer: can be implemented now, while we think whether add the parallel: option or not. I'd love to hear opinions from other maintainers/users on this.

[smyrman]

With normal shell syntax, tasks can be sent to the background today with &, which is basically parallel execution, but with no rendevouz. if they don't require cleanup, or can be easily terminated through commands such as kill, that might be good enough:

# Example file from the top of my head, may contain some errors.
default:
  cmds:
    - task: connect_ssh_tunnel
    - sleep 1
    - curl http://localhost:9999
    - tass: disconnect_ssh_tunnel

connect_ssh_tunnel:
  cmds:
    - ssh -L 2000:9999 somehost&

disconnect_ssh_tunnel:
  cmds:
    - ps -ef | grep 9999 | grep -v grep | awk '{print $2}' | xargs kill

If parallel execution is to be included as a feature in task, it shoud bring something more to the table than what can already be achieved by &. E.g. rendevouz (explicit wait for completion) or the ability to (gracefully) terminate the task, e.g. by retrieving it's PID. If it doesn't bring any of these features to the table, I don't thing a parellel flag gives us much.

[smyrman]

I see the references commits already adds something that looks like rendevouz (WaitGroups), so then I assume it's already probably more useful ;-) Just trying to highlight what I think might be worth considering for parallel executions, not discouraging it.

[sascha-andres]

The idea is to capture the ouput of all commands, so I can dump them ( so they will show up in git push output ). If one of the commands fail, the execution of task returns a non zero code and push fails.

[andreynering]

To have a definition, I've decided against a parallel: option for now, since the exactly same use case can be solved with cmd: in deps.

I can change my opinion in the future, specially if more users ask for it, or have a different use case.

@sascha-andres If Task work well for you as a Git hook, it would be interesting to have an example in go-task/task repo.

[sascha-andres]

The preparations ( eg get project configuration, create a local clone ) that are dependencies for a task may not run in parallel as they depend on each other. We might wrap those in a single prepare task and have that run as a dependency unless one task has another dependency ( because for example build times for Go are negligible ).

For each language that is used ( Go, PHP, Java, .NET, ... ) I wanted to have a Task that executes commands in parallel and have no worries about dependencies. That boils down to something like this:

prepush:
  cmds:
    - getconfig
    - localcheckout ( setting a variable to the path )

cleanup:
  cmds:
    - remove localcheckout
    - write history

golang:
  deps:
    - prepush
  parallel: true
  cmds:
    - lint, etc
    - task: cleanup // with defer: `defer: task: cleanup` at the beginning

php:
  deps:
    - prepush
  paralell: true
  cmd:
    - ....
    - task: cleanup // with defer: `defer: task: cleanup` at the beginning

The hook would have been just task golang. So while for a software build running dependencies in parallel and the build steps one after another in this use case the requirements just switch.

Above theoretical TaskFile is just to have a more expressive explanation for why I made this proposal in the first step.

I am fine with not including parallel into task at this time and to prevent clutter I will remove the branch from the repo. The complexity and amount of code to provide a base for discussion is/was not big and reproducible in a short amount time.

[andreynering]

@sascha-andres Hmm... with your example I now better understood the use case. In fact it would be cleaner when you also want to use deps as dependencies instead of commands.

Each possible solution has its drawbacks. Here's another possible syntax:

golang:
  deps:
    - prepush
  cmds:
    - go: {task: cleanup} # or "async: {task: cleanup}"
    - lint, etc

Let's keep this open for now for reference and discussion.

[smyrman]

If we could solve #53, The following could be an option:

golang:
  desc: run golang linters
  deps:
    - golang/lint1
    - golang/lint2
  cmds:
    - golang/cleanup

golang/lint1: # example name, replace lint1 with real binary name
  deps:
    - prepush
  cmds:
    - golint-cmd1

golang/lint2:  # example name, replace lint2 with real binary name
  deps:
    - prepush
  cmds:
    - golint-cmd2

Then it's also possibly to explicitly run golang/lint1 and have then dependencies resolved.

[smyrman]

For the defer syntax, what about

```yaml:

• task: name
  defer: true

• sh: command
  defer: true
  `` And allow theshandtaskprefix incmds`. Then existing structures can be reused.

[andreynering]

@smyrman

If we could solve #53, The following could be an option

Sure. There's a couple of other issues that needs work. 😟

For the defer syntax, what about

Definitely an option. Or we'd need to allow something like this:

- defer: {task: foobar}

[smyrman]

Sure, either syntax would work, but I suppose the boolean flag one would be slightly easier to parsed directly into the cmd struct:

https://github.com/go-task/task/blob/55672410cde75ea46b593f56ad8b4efdbb4c8e82/command.go#L9

I suppose you can still do that with a defer: {...} prefix, just it's slightly more work.

[sascha-andres]

@andreynering @smyrman I added a branch using functions as an indirection for running tasks, a task and dependencies. See SHA: c11113de3765b82ea2fa8a34b5b8fe410845aa3c

This would make it possible to use go-task as a library and overriding the default behavior. Maybe we can get around discussions different execution models at the root level ( by that I do not mean never implementing a different execution model )

Update: added a commit providing With* functions

[andreynering]

Please join #task room in Gophers Slack. Let's continue the conversation there, so we don't create a lot of noise here. :wink:

[smyrman]

@andreynering would you like to create an early PR for easier feeback & discussions? Just mark it with "WIP / DON'T MERGE".

It might be easier to give feedback tom some of your code in-line.

[andreynering]

I decided to close some old issues

While some ideas discussed here are valid, let's keep things simple for now

[frederikhors]

I do not understand if this is possible today. @andreynering Is it?

[andreynering]

@frederikhors All dependencies listed in deps: will be called in parallel. Only after they return, the commands in cmds: will run. That's the way of running things in parallel today.

This issue is about an alternative approach, but I decided against it for now.