Gulp: Update Gulp 4 recipe for splitting tasks between multiple files

[bradcerasani]

TL;DR - Define composite tasks in gulpfile after gulp.registry(hub)

---

Split tasks across multiple files has been updated to use the new gulp-hub module, but with the Gulp 4 registry forward referencing tasks is no longer possible.

While sequencing is still being discussed in #972 and (prior to that) #802, I've created a test repo with a basic example of splitting tasks across multiple files with a slightly modified recipe.

https://github.com/bradcerasani/gulp-4--split-tasks

Description

Standalone tasks are defined in the tasks directory, and gulp-hub is used to create a new registry. Tasks that use a combination of sub-tasks are now defined in the gulpfile, after gulp.registry

While this will require a minor restructuring of tasks in some projects, I actually prefer defining composite tasks like default, watch, and build in my gulpfile and using the tasks directory strictly for standalone tasks. In larger gulpfiles composite tasks are typically short and called far more frequently than their standalone siblings, so IMO it's easier to reference them in one place.

File structure (unchanged):

gulpfile.js
tasks/
├── clean.js
└── scripts.js

gulpfile.js

'use strict';

var gulp = require('gulp');
var HubRegistry = require('gulp-hub');

// load some files into the registry
var hub = new HubRegistry(['tasks/*.js']);

// tell gulp to use the tasks just loaded
gulp.registry(hub);

// define composite tasks
gulp.task('default', gulp.series('clean', 'scripts'));

If this is a viable solution, I'm happy to submit a PR with an updated recipe.

[phated]

I think this is a last-resort suggestion, but let's see how https://github.com/gulpjs/gulp/issues/972#issuecomment-93402690 pans out first.

[mikestopcontinues]

@phated Why does Undertaker check for task existence on registration rather than on run? That's really the sticky bit, isn't it? For some composite tasks, it's alright to use the gulpfile as @bradcerasani suggests, but not always so. Pretty much every task depends on something else, even if it's just a "clean".

[NekR]

@mikestopcontinues indeed, just got into this issues after updating/trying gulp4. My simple task-generation utility generates clean task in the after all other task, but some of them depends by clean, so it failed even at first run. It's not a problem to move clean definition to top, but it's just uncomfortable and strange. @phated Is it just WIP artifact?

[jmm]

@mikestopcontinues

Why does Undertaker check for task existence on registration rather than on run? That's really the sticky bit, isn't it?

I submitted a proposal for supporting "forward referencing" in #1038. Just putting a placeholder function for registered tasks in series|parallel composed functions and proxying to the real function at task runtime might be a simpler implementation, but I think that would restrict it to a system where only the last registered function with that name is used in all cases. I based my implementation on the premises discussed in #972, such as that a registered task name can refer to different functions at different times, e.g. passing the same task name string to different calls to series ()|parallel () can reference different functions if the task name is overwritten with a different function by a call to gulp.task() in between.

[mikestopcontinues]

I think giving precedence to the last-registered function offers much greater flexibility. Even with a few hundred tasks, it's easy to keep names from colliding unintentionally, but task overrides allow for what I do at work.

I'm in charge of a bunch of similar apps. They all inherit from a common repo of gulp tasks, but each overrides a handful of tasks to account for the particular needs of that project. Using a different unit testing library or watching an additional directory during dev. Most of the overridden tasks are deep in the task dependency tree.

Is there something I'm missing that makes task name reuse worth it?

[jmm]

@mikestopcontinues

Is there something I'm missing that makes task name reuse worth it?

I can't provide any insight into that. @phated would be the one to ask. I suggest you post in #972 about the use case you're describing. I'd say that merits discussion while this stuff is still being hashed out. You ought to be able to do this as a workaround anyway:

function last (task) {
  return function (done) {
    return gulp.task(task).apply(this, arguments);
  };
}

gulp.series('A', last('B'), 'C');

[mikestopcontinues]

Thanks, I'll post there, @jmm. Likely, I'd use a homebrew .series and .parallel that postponed seeking the tasks until runtime, or at least until all tasks were registered.

[phated]

@mikestopcontinues I've actually been thinking about your use case _a lot_ and I think it should be solved with a custom registry (recently documented at https://github.com/gulpjs/gulp/blob/4.0/docs/API.md#gulpregistryregistry) but with a configuration object that allows overrides of tasks. e.g.

var util = require('util');

var DefaultRegistry = require('undertaker-registry');

function MyCompanyTasksRegistry(opts) {
  DefaultRegistry.call(this);

  opts = opts || {};

  this.set('clean', opts.clean || function(done) {
    done();
  });
  this.set('css', opts.css || function(done) {
    done();
  });
}
util.inherits(MyCompanyTasksRegistry, DefaultRegistry);

module.exports = new MyCompanyTasksRegistry();

[mikestopcontinues]

@phated, I really appreciate you putting thought into my case, though I just don't understand the reasoning behind the current implementation. As it stands, I'd rather just monkey-patch .series and .parallel for lazy task lookup because it'll also allow me to load a directory of tasks, which is another big convenience of how I do things now. But maybe I'm just missing something? I'd love to hear your reasoning for early task lookup rather than at runtime.

(My own gulp3-based .parallel and .series functions accept regex objects for lookup at runtime. This way, my build task does /build..*$/, which for some projects matches the global "build.css" and "build.js", but in particular projects also matches the local "build.font" or "build.html" in their local tasks directories.)

[phated]

Figured I would update this post. It was easiest to implement forward references as a custom registry - see https://github.com/undertakerjs/undertaker-forward-reference - as it relates to gulp-hub, I am going to suggest to that project to inherit from the linked custom registry.

[mryellow]

Breaking this with v4, makes Gulp much less organised, monolithic config files with mountains of functions. This isn't an improvement.

[contra]

@mryellow Yeah, which is why it's a major version bump (v3 -> v4) - there are breaking changes. Things will break. Things will update and be fixed for the new changes. Such is life - its unreleased, and will be fixed before the release.

[mryellow]

It's not so much the breaking changes, as "I can no longer make my task manager track task dependencies across files".

gulp-hub can be used, but means adding the registry in each file containing tasks. Although not too verbose, seems like unnecessary overhead.

Example multi-file use-cases are one layer deep, with a master gulpfile including a directory of tasks which don't depend on each other. Any interdependence moves up a layer to the including gulpfile, where tasks from any of the files can be used together.

Given: The new way is semantically potentially much more robust and while the original way is rather open to problems with it's adhoc "structure".

The issue remains that the current layout for 4.0 means defining complex interdependent tasks becomes more verbose and harder to separate into short easy to read task files. For many projects this will either mean a complete re-structure of tasks, or loading gulp-hub registry for each file. For most other projects it will be a simple migration, but not for all.

There are problems with the old way, but it makes things much easier, which is a great selling point when it comes to devops.

[phated]

Please stop spamming our projects. An upgrade path has already been developed.

[zheeeng]

gulp-hub requires gulp 3.8+ but, gulp 3.8 has no .registry method