Gulp: Theoretical question about forking a stream in gulp

[gionkunz]

Hey there. I'm relatively new to gulp and streams but I have the following situation. I have two pipelines A and B where A and B both have 5 steps in the pipeline where the first 3 are the same. Theoretically and from an efficiency standpoint it would make sense to use the same pipeline for the first 3 steps and then fork the pipeline into two separate ones.

I'd think of something like this:

gulp.src('./src/*.js')
  .pipe(sharedOne())
  .pipe(sharedTwo())
  .pipe(sharedThree())
  .pipe(gulp.fork(function(a, b) {
    return eventStream.merge(
      a.pipe(aOne()).pipe(aTwo()),
      b.pipe(bOne()).pipe(bTwo())
    );
  }));

I'm sorry if I don't make any sense or if there is already a known stream utility like this but as I sad I'm relatively new to this.

Any help would be much appreciated.

[heikki]

This recipe looks related: https://github.com/gulpjs/gulp/blob/master/docs/recipes/sharing-streams-with-stream-factories.md

Does that help? :)

Support questions http://stackoverflow.com/questions/tagged/gulp

[gionkunz]

Well, partially yes :-) The Problem lazypipe (didn't know it yet, thanks for the hint!) solves is to reuse predefined pipelines. However, if I want to fork a pipeline I'd need to create a stream reader that buffers and offers N new stream readers for N instances of the fork.

              --> X --> Y --> Z
             / 
A --> B --> C
             \
              --> I --> II --> III

As far as I understand streams they can only be read by one reader who changes the stream position. The use-case of forking a stream would mean that the a reader would buffer and create N streams for other readers to read further.

A forking module would basically clone the stream (or buffer and provide different positions for each reader).

Am I missing something?

[gionkunz]

A use-case for this would be that I have several JavaScript pre-processing steps in a shared pipeline (lets say linting, 6to5 and testing), then I fork the pipeline where one fork would transform the stream into a browser version and the other one into commonJS.

[gionkunz]

Of course I could run both tasks in parallel but there would be an efficiency loss as I run the shared steps twice.

[nfroidure]

@gionkunz a tip i used for gulp-iconfont and it's low level gulp plugins is adding a clone option https://github.com/nfroidure/gulp-iconfont/blob/master/src/index.js#L15

That way, i can reuse the source ttf file along the all pipeline (ttf2eot, ttf2woff) and still get it as part of the outputted files. For that matter a 'fork' or 'clone' plugin could have make sense indeed.

[heikki]

There's gulp-clone. Maybe that's part of the solution?

[gionkunz]

Thanks I'll check gulp-clone. I've also found https://github.com/deoxxa/fork-stream/blob/master/index.js but it's limited to two forks which is not necessary I think. I'll research more, but if nothing fits my needs I'll pitch for a gulp-fork :-)

[nfroidure]

gulp-clone should work imo but i think there is some improvements you could do, first by removing this https://github.com/mariocasciaro/gulp-clone/blob/master/index.js#L17 since file.clone() work with streams (and null files also).

[nfroidure]

Also contra's issue is waiting for your PR ;) https://github.com/mariocasciaro/gulp-clone/issues/2

[popomore]

Is it an usecase of gulp-mirror?

[gionkunz]

@popomore , Yes!! :-) Thats it I believe. Why do you instruct to do npm install gulp-mirror -g ?

[nfroidure]

@popomore it seems that the gulp-mirror approach doesn't allow you to reuse the mirrored streams in the same pipeline without concurrency issues (see https://github.com/sindresorhus/gulp-filter/issues/29 ).

In fact, you could meet the same kind of problems with the gulp-clone current implementation but according to its API it can be fixed (see https://github.com/sindresorhus/gulp-filter/issues/29#issuecomment-73474426 since the API is very similar) while it is not simply feasable with gulp-mirror.

[nfroidure]

No problem using gulp-mirror mirrored streams as simple readable streams though.

[gionkunz]

A repository with a README containing known and useful streaming modules would be awesome... It's so hard to find the right terminology for such abstract problems.

[popomore]

@gionkunz that's my init template for my every repo. I will update README about your usecase.

[gionkunz]

@popomore , yeah maybe it should not need to be installed globally right?

[heikki]

gulp-clone

var gulp = require('gulp');
var clone = require('gulp-clone');
var es = require('event-stream');
var concat = require('gulp-concat');
var rename = require('gulp-rename');

gulp.task('clone', function() {
    var all = gulp.src('src/**/*.js').pipe(concat('all.js'));
    var a = all.pipe(clone()).pipe(rename('browser.js'));
    var b = all.pipe(clone()).pipe(rename('common.js'));
    return es.merge(a, b).pipe(gulp.dest('dest'));
});

gulp-mirror

var gulp = require('gulp');
var mirror = require('gulp-mirror');
var concat = require('gulp-concat');
var rename = require('gulp-rename');

gulp.task('mirror', function() {
    return gulp.src('src/**/*.js')
        .pipe(concat('all.js'))
        .pipe(mirror(
            rename('browser.js'),
            rename('common.js')
        ))
        .pipe(gulp.dest('dest'));
});

--edit

Fixed gulp-mirror example (maybe)

[gionkunz]

Thanks guys. All your input was very helpful!

[heikki]

:beers:

[nfroidure]

@heikki the gulp-mirror example is false according to the doc. It says the mirrored streams must be used elsewhere.

Also through.obj().pipe(rename('browser.js')) is the same than rename('browser.js') since pipe returns a ref to the piped stream.

Would have use it like that according to the doc :

var gulp = require('gulp');
var mirror = require('gulp-mirror');
var through = require('through2');
var concat = require('gulp-concat');
var rename = require('gulp-rename');
var combiner = require('stream-combiner');

gulp.task('mirror', function() {
    var commonStream = through.obj();
    var browserStream = gulp.src('src/**/*.js')
        .pipe(concat('all.js'))
        .pipe(mirror(commonStream))
        .pipe(rename('browser.js'));

      return combiner(
        browserStream,
        commonStream.pipe(rename('common.js'))
      ).pipe(gulp.dest('dest'));
});

[heikki]

@nfroidure Thanks, rename change makes it simpler and it seems to work identically to gulp-clone version. But.. where do you see that docs mention? I'm looking at this https://github.com/popomore/gulp-mirror#usage

[popomore]

@nfroidure The approach of gulp-mirror and gulp-clone is same, but the implementation is different.

gulp-mirror can create many mirrored streams(not only two), these streams pipe to output without end, after all streams are ended(listen end event), call the end method of output.

The mirrored stream can be transform stream(not only be readable stream).

[popomore]

Should be work in this way, I can write an example.

gulp.task('mirror', function() {
    return gulp.src('src/**/*.js')
        .pipe(concat('all.js'))
        .pipe(mirror(
          rename('browser.js'),
          rename('common.js')
        ))
        .pipe(gulp.dest('dest2'));
});

[heikki]

@popomore Real world usage example in the docs would be good. I did try one and got unexpected results (probably because I have misunderstood something).

[popomore]

Hey, here is an example.

https://github.com/popomore/mirror-example

[nfroidure]

Well, it seems like I completely misunderstood the usage :D. It looks like you can just put one passthrough stream to gulp-mirror and i think it is completly useless without using a duplexer or a lazy pipeline.

Maybe something like that could help to get it faster:

var streamA = through.obj();
var streamB = through.obj();

gulp.src('**/*')
  .pipe(mirror(
    duplexer(streamA, streamA.pipe(transform1()).pipe(transform2())),
    duplexer(streamB, streamB.pipe(transform3()).pipe(transform4()))
  )
  .pipe(gulp.dest('.'))

[popomore]

Indeed, but i prefer multipipe, the stream flow is clear.

var pipe = require('multipipe');
var sharedSteam = through.obj();
gulp.src('**/*')
  .pipe(sharedSteam)
  .pipe(mirror(
    pipe(transform1(), transform2()),
    pipe(transform3(), transform4())
  )
  .pipe(gulp.dest('.'))

[nfroidure]

Agreed!

[heikki]

^ Let's move further gulp-mirror discussion there.

[popomore]

Actually, gulp-mirror is inspired by multipipe, but the stream flow is different.