Task: Add "imports" or "includes"

[JulienBreux]

Hi,

Request

Add the possibility to use import or includes to imports some others tasks.
With the default strategy as namespace.

Example

File: docker/v1.0/Taskfile.yaml

version: "2"

task:
  build:
    cmds:
      - docker build...

File: go/v1.0/Taskfile.yaml

version: "2"

task:
  build:
    cmds:
      - go build ...

File (in my project): Taskfile.yaml

imports:
  docker: https://.../docker/v1.0/Taskfile.yaml
  go: file://.../go/v1.0/Taskfile.yaml

— or —

includes:
  docker: https://.../docker/v1.0/Taskfile.yaml
  go: file://.../go/v1.0/Taskfile.yaml

task:
  build:
    - task: "docker:build"
    - task: "go:build"

How to use?

task go:build go:docker

— or —

task build

Finally

What do you think?

[smyrman]

This is a duplicate of #54, but I think the description here is very clear, especially with the namespace suggestion. After the releasee of task 2.0, it's defiantly possible to add.

I also want to point out that the file:// use-case is _possible today_ by relying on calling the task binary with relevant input from the relevant directory. It has some short-commings, e.g. if task is not in PATH, but we are successfully using it for many of our v1 Taskfiles.

tasks:
  docker/build:
    dir: ../go/v1.0/
    cmds:
      - task build VARX=something

[smyrman]

54 closed in favor of this issue

[smyrman]

Copied labels from closed ticket.

[JulienBreux]

@smyrman A big thx for this tool. Do you need help or time?
:)

[smyrman]

You should thank @andreynering, he's the main author:-) My contributions are relatively modest. Any help in improving the tool is most welcome!

[JulienBreux]

Okey @smyrman. And big thx to @andreynering :)

[andreynering]

@JulienBreux Glad you found it useful!

Help on this would be welcome. Let us know if you plan to work in this.

Something we should discuss: when executing a task defined in an imported file, should this task run in the current directory or in the imported file's directory? Or have an option to allow both? :thinking:

[JulienBreux]

Humm, good question, but I really think that is in the current dir.

The idea is to "please, import tasks in the current dir".

In my example, it's very useful to import some tasks in the current dir to be DRY cross repos.

[andreynering]

@JulienBreux Ok, that makes sense as a first step. But maybe in the future we should think about allowing executed in the imported file dir.

[josdotso]

This may help: https://github.com/hashicorp/go-getter

[andreynering]

I plan to implement this soon

I think I'll only allow importing from files at first, and thought about others later

[JulienBreux]

👍🏻

[andreynering]

Implemented on master, see documentation here: https://taskfile.org/#/usage?id=including-other-taskfiles

For those that still have interest in this feature, if possible, do a quick test and tell me if you find any bugs or if this isn't enough to solve your use cases. I plan to wait one or two weeks before releasing it.

[andreynering]

As a side note: I decided not to allow recursive import (i.e. included Taskfiles are not allowed to include other Taskfiles) for the sake of simplicity.

If anyone think that would be useful, please open another issue for that.

[JulienBreux]

@andreynering thanks a lot for that.

But a tiny question: Is it possible to use "network" protoc'? (eg: https://...)
The primary idea was the Taskfile portability :)

[andreynering]

@JulienBreux As I said above, I decided to implement importing files first time have something working.

Could you please open another issue explaining with more detail how you see that working?

For example: how would Task know when to re-download the file, since it won't know when it has changed? (I'm assuming we want some caching mechanism, to not make running a simple command pretty slow).

Or perhaps it makes sense to have a task --fetch command for that?

In other words: I want more opinions on how that should work.

[smyrman]

Sorry for responding to this ticket before a new one has been created, but I can answer some of those questions. Feel free to copy-paste it to the new ticket if you choose to create one:

For example: how would Task know when to re-download the file, since it won't know when it has changed? (I'm assuming we want some caching mechanism, to not make running a simple command pretty slow).

Luckily we don't need to reinvent the wheel on this one. The HTTP protocol defines a few things you could use.

For GET and HEAD requests, you can add an If-None-Match hear with the E-tag value of the last response:

• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match

There is also the If-Modified-Since request header that could be used with the value of the Last-Modified header when an ETag is not available in the previous response:

• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since

When none of these headers are present, then fine, don't cache anything.

The cached files would need to go in the .task folder I suppose, together with E-tags and Last-Modified headers when available.

[JulienBreux]

Could you please open another issue explaining with more detail how you see that working?

Very strange to open another issue, no? This issue was very explicit:

includes:
  docker: https://.../docker/v1.0/Taskfile.yaml
  go: file://.../go/v1.0/Taskfile.yaml

But okey, no pb 😃

In other words: I want more opinions on how that should work.

Yes, it's possible to play with cache? I'm agree with @smyrman.

[smyrman]

Feel free to copy-paste it to the new ticket if you choose to create one

To clarify, what I meant with this was: if the feature is still desirable, please create another issue, and feel free to cop-paste my comment.

Splitting the remaining work into a new issue allows the discussion to be focused around the missing functionality, making it easier to search for and digest for newcomers. It's nice if you can create the new ticket @JulienBreux, because it validates that the remaining parts of this issues is still desirable.

[andreynering]

Sorry

Actually keeping this open since it's easier and discussion is happening here anyway

[andreynering]

About the cache headers, well, I agree that it helps, but I'd prefer not to make a HTTP request most of the time.

[tonobo]

Rubocop also introduced a kind of caching which really sucks when you're often update the rules. The rubocop keeps the downloaded file within the git directory and uses the "cached" version for the next 24h.

I don't like this, but i also know the downsides pulling the content every time. Maybe there would be an option to configure this.

[tonobo]

I've added added the includes stuff now in a fork, i try to keep them up to date with the master and keep it mergeable. Please see https://github.com/tonobo/task

I've added also a kind of interactive shell mode, the commit should be pickable if you wan't to https://github.com/tonobo/task/commit/4fbe166d6ebc130baa7ac4ae049dfd1f3c8e1201

But basically there shouldn't be any breaking changes.

Docs:
https://tonobo.github.io/task/#/usage?id=including-other-taskfiles
https://tonobo.github.io/task/#/usage?id=interactive-shell-mode

[thirdeyenick]

As you basically implemented this in your fork, would it be possible to create a pull request for it? Including Taskfiles via http or even git is a very useful feature I'd say.

[tonobo]

I've done quite more changes which might not considered stable and this project aims for stability over features. The pull request has been closed a while ago, but is referenced.

[andreynering]

Hi @thirdeyenick,

I decided to hold this feature for a while. That doesn't mean it'll never come, but it means I think other stuff should be addressed first.

You can see my comment here.

If you ever try @tonobo's fork, I'd still like to know whether that implementation works fine for you, etc.

[tonobo]

I've added git clone support for includes as well, but it's actually done by a shell out for several reasons. By doing this with libgit or golang native git bindings I always need to handle the whole ssh stuff at my own and this really sucks. Because it isn't only reading and opening a few keys, you need lookup for the keys on the ssh-agent and completely re-implement what git already does. @andreynering What do you think about it? Would you merge it, just for interest.

[andreynering]

@tonobo Not sure. GitHub allows to download any file through HTTP, so in theory we don't need to add Git clone support.

We could have a shortant if necessary (that would be converted to the right URL):

includes:
  foobar:
    github: go-task/task
    file: foo/bar/Taskfile.yml

This is just a braindump, though, not a final decision.

Doing these includes stuff in steps would allows us to reduce the impact to the codebase.

[marco-m]

I would like to add another caveat to the idea of including remote taskfiles: they break repeatability.
To elaborate on this:
say I have the root taskfile in a git repo and I checkout a given commit. With remote taskfiles, each time I run task I could get different results, because the remote taskfile escapes version control. They are a bit like latest of a Docker image: convenient, yes, but dangerous (can change / can break without warning).
To me, remote taskfiles should be provided as git submodules (or equivalent), providing full dependency tracking and repeatability.

[JulienBreux]

I think that's very important to add checksum!

For example:

imports:
  docker:
    url: https://.../docker/v1.0/Taskfile.yaml
    checksum: 72c522ea01264d7adbbb851292b1bd7d
    cache: 1d
  go:
    file: .../go/v1.0/Taskfile.yaml
    checksum: e4f8911c284dd6d322b390c4f3db9e2a
    cache: 1y

It's very helpful for managing the cache. And in the case where you don't use checksum you can use cache (duration) to determine the duration.

[marco-m]

Yes, checksum and cache make the situation slightly better, but consider:

• checksum doesn't allow to go back in time to a previous version. If in the repository containing the root taskfile I check out a previous git commit, which contains a previous checksum for the imported taskfile, running task will fail.
• cache, at least with relative values as in the example (1d, 1y), is not repeatable, it is set the first day that the taskfile is run. If we share the repo and you run task today and I run task tomorrow, the two of us will have a failure on different days.

[JulienBreux]

cache, at least with relative values as in the example (1d, 1y), is not repeatable, it is set the first day that the taskfile is run. If we share the repo and you run task today and I run task tomorrow, the two of us will have a failure on different days.

Yes, but for a stronger immutability and repeatability I suggest to use a single Taskfile.yml ;)

[marco-m]

Il me semble que on fait les Shadoks nous deux 😄

[andreynering]

I overall agree with @marco-m.

That's part of why I decided not to do this without more thought: One can already have "remote" Taskfiles, but need to vendor it. i.e. create a script that use curl to download the correct Taskfile, and actually commit it to the repository. This will keep you safe for the future.

Also, do any other build tools support something like this? I don't think so.

[smyrman]

Terraform, although not exactly a build tool, supports importing remote _modules_. For Git, a git checkout reference can be used to pin to a specific tag or branch.

• https://www.terraform.io/docs/modules/sources.html

Don't personally feel it's worth putting an equal amount of work into remote Task files, or supporting it at all, but peaple may use the tool in different ways. If there is such a case, inspiration (or even code perhaps) may be taken from Terraform project, I suppose.

[marco-m]

@smyrman yes, terraform supports remote modules, but with the following characteristics:

• they are versioned
• the master tf file can optionally specify the version
• if no version is specified, then latest is obtained

but the whole idea of tf remote modules is that ALL versions are always available. It is not enough to put a file on an HTTP server :-) Said in another way, tf remote modules are similar to a directory with all releases of a package or to a coarse git repo.

[smyrman]

It is not enough to put a file on an HTTP server :-)

Actually, it sort of is, as long as you are comfortable with delegating the responsibility of handling versions completely to the HTTP server. Even with Git, you can force-push, delete repositories, rewrite tags and more. Without a registry, there actually is no way to _gurantee_ that versions stay available, nor that they change, so the best you can do is really just validate that the URL is serving the same content as it did the last time you checked (and calculated an MD5 or SHA1).

Even if task imports are _not useful_ without versioning, that doesn't really mean that versioning should be solved _by_ Task. Solving versioning is a complex task.

As an example of a HTTP server that handles versioning, Github have many ways of letting you refer to a specific version of a file via HTTP, such as replacing "master" in the raw URLs with a tag or specific SHA1, or by adding Taskfiles to a release.

That said, I am not really advocating that the use-case for adding support for remote includes is a strong one. If it is a strong case for others, an important thing to clarify is which content one would want to be able to download:

1. Downloading publicly available content (someone sharing a Taskfile with the world).
2. Downloading private content/repositories, e.g. via Git.

If 1 is the target use-case, then HTTP(s) download with (optional) MD5/SHA1 checks should be fine. If however 2 is the target use-case, than you may want to choose an approach with direct Git integration instead so that authentication and content validation can be outsourced to Git (like Terraform git URLs perhaps). Or, of course, as already suggested, let the user write their own tooling for managing private dependencies.

[marco-m]

It is not clear to me if we are actually saying the same thing or if we are misunderstanding each other.

Even if task imports are not useful without versioning, that doesn't really mean that versioning should be solved by Task. Solving versioning is a complex task.

This is the whole point of my comments in this thread.