Rxjs: Improved build process

[benlesh]

So because of a tweet, there is some interest in improving the build process for this project.

I'm game, but we need to identify requirements for something to be viable as a replacement for what we currently have with npm run scripts.

Required:

1. It cannot be slower than the current build process.
2. It must be easy to maintain and extend. (This is subjective, sure, but I think most of us can make _fairly_ objective decisions here)
3. Must have exactly the same output in terms of what lands in the dist/ directory.
4. Cannot change our testing suite again.
5. Cannot completely restructure our codebase for the sake of the build script.
6. Must run cleanly with solid console output in Travis.
7. Ability to build pieces of our distribution one at a time (CJS vs ES6 vs AMD, etc)
8. Should build on Windows

9. Needs to have a solid upgrade path for newer versions of TypeScript

   Nice to have:

10. Faster development time builds. Things like aborting the previous build and starting a new one if you save a file. Maybe only building what needs built

11. Flexibility for when we split our code base into separate packages (moving WebSocketSubject to an rx-dom type package, for example)
    

    Why aren't we (meaning, mostly me) spending more time on this?

Mostly because what's in place works, and there are more important things to deliver on with a limited number of contributors.

[benlesh]

It should also build on Windows: #1601

[Igorbek]

gulp #1592?

[benlesh]

@Igorbek way back when this project started, gulp-typescript wasn't an option because it didn't support the version of TypeScript that we were using. We'd need to confirm that it supports 1.8.10, with a solid upgrade path to 1.9 and higher.

[benlesh]

... in fact, I've just updated that as a requirement.

[d3viant0ne]

@Igorbek - This popped up on twitter originally and the discussion about npm scripts vs. gulp vs. something new. I'm totally up for helping out here but imo the best result is going to be the mix of gulp / scripts the produces the optimal build workflow for the rxjs devs.

@kentcdodds had also mentioned interest in some sort of middle ground solution / pattern, I think his input here would be invaluable.

@blesh - I'm using gulp-typescript on 1.8.10 without issue

[kentcdodds]

Oh... So... 😅 , I spent the day to build a solution for this (mostly for fun). Thanks for mention @d3viant0ne!

This is still a bit of a work in progress, but I think that this could really help things here:

https://github.com/kentcdodds/p-s

Feedback is definitely welcome.

[d3viant0ne]

@blesh - My first question after that twitter thread was, do any of the rxjs contribs have any gripes with the current build system?

I personally agree with @sindresorhus in regards to the build system, i'm not the one working on the project. Point being, in reality if it isn't broken in the eyes of the contribs, does it really need fixing?

[fyockm]

Might I recommend scripty?

[kentcdodds]

Unless I'm mistaken, this wouldn't solve the problem of having several dozen lines of scripts.

[fyockm]

no, it would not. but that was _not_ one of the requirements...

[kentcdodds]

I realize that's not one of the requirements so feel free to ignore. It's the reason I'm not using it and built p-s. Just thought I'd mention that.

[benlesh]

The only real requirement is that whatever the new solution is, it's fundamentally _better_ than what exists.

[d3viant0ne]

@fyockm - For reference, the dozens of lines of scripts is how this all came about in the first place.

@blesh - Next question, Do you want this with rxjs 5 or after it's out of beta for less churn?

[kwonoj]

I've shared my opinion in https://github.com/ReactiveX/rxjs/issues/1592, would vote to npm scripts than other task runner. I know there's lot of different opinion for this, just my 50¢. I just don't see strong benefit by introducing task runner with its own complexity of writing configurations.

[trxcllnt]

gulp is not my cup of tea. better-npm-run is pretty fresh. combining with parallelshell would probably speed things up.

[d3viant0ne]

@trxcllnt - Thanks for the suggestion, I just started tinkering with better-npm-run and i'm liking it so far. And for reference, i'm good with gulp but I don't just want to default to it. Like all things in life, somewhere in the middle lies the right answer.

[kentcdodds]

combining with parallelshell would probably speed things up.

Just FYI, p-s supports running scripts in parallel :-)

[d3viant0ne]

@kentcdodds - I'm playing with it now. Converting something gulp heavy to see how much code I can remove

[tetsuharuohzeki]

I'm favor parallelshell approach than gulp & gulp plugins based system.

gulp can manage to task dependency and task concurrency almost nicely, however, gulp runs on a single process so it would be a bottleneck, and gulp-plugin ecosystem requires us to use indirect dependencies.

[staltz]

My opinion is that the build system we have right now with npm scripts is fine. It's "big" because there is essential complexity, but when you need to tweak some task or add a task, it's a no-brainer. As an RxJS maintainer myself, these npm scripts have never been a headache to me, but I'm certain gulp would be (as it has been in the past for me).

I don't see any gain in moving the build system to something else just because someone on Twitter has an opinion. Look at it from a practical perspective: me and kwonoj are the "housekeepers" here and we would prefer npm scripts.

Windows users: I'm sorry, just use a Virtual Machine running an OS with proper bash or try to get Windows' own bash working. Don't look at this as an insult, I did this myself for a couple of years. I had a Windows machine and just installed Ubuntu as a virtual machine to get things done.

[kentcdodds]

@staltz, I should preface everything I say with: "I'm not a contributor so don't listen to anything I say. I'm not allowed to have an opinion here" :-)

Of I were in your shoes, I'd probably stick with npm scripts too. I built p-s to see what it'd look like and because I know some people would appreciate it. I'm not 100% convinced that I'll actually use it in any of my own projects because I find npm scripts good enough.

That said, there are definitely tools that you can use to make your scripts more Windows user friendly if you're interested in lowering the barrier of contribution (which I expect @blesh is). I'd link these for you but I'm on mobile so forgive me:

• rimraf
• cpy-cli

Just looked through your scripts and I think using those two packages will get you most of the way there for supporting Windows better. Along with double checking that scripts are properly escaped (I believe globs need to be double-quoted).

Good luck y'all!

[david-driscoll]

I'm fine with the tasks as they are, at first I thought "god this is weird" but it's slowly grown on me, in that it's simple and straight forward. We don't have to deal with more advanced cases, we're not making a big large application. We're making a simple library, as long as the builds suit the needs of the consumers with cjs, amd, etc, then what is the harm?

The only annoying thing I've noted is that since tests rely on the compiled library, you have npm run build_cjs then npm test. I'm sure this flow could be modified with a daemon of some sort.

Personally I develop on windows full time, and the _only_ issue I've ever seen is the occasional error from tslint that slips through because the tslint cli doesn't support globs on windows.

I do ensure that I have C:\Program Files\Git\usr\bin in my PATH, as it contains utilities like rm.exe which is a windows compiled binary. I don't run the git bash shell, cgywin or anything special. I do use Cmder over the default windows console.

[fyockm]

On the heels of @kentcdodds, I would also propose using commands from shelljs to make things a little more Windows friendly. It contains many of the Unix style commands for use across platforms in one library, for whatever that's worth.

[staltz]

I'm totally favorable of using rimraf, mkdirp etc to try to make npm scripts work better in Windows :+1:

[d3viant0ne]

@staltz - I don't think anyone is looking to change things for the sake of doing so, though that is pretty much how the thread on twitter started.

The requirements are pretty clear and no matter what the tooling preferences are, It's hard to argue that there isn't room for improvement in the realm of development experience / general usability.

Nobody is suggesting going all "gulp-tastic" but if improvements can be made while picking up some of the nice to haves, how is that not a net gain? In particular when the people raising their hands to do it are not people who contribute to the project already.

[alexeagle]

FYI angular team is also evaluating build systems at the moment.

[d3viant0ne]

@alexeagle - Is there an issue open related to the discussion? I'd be curious to see the discussion / proposed options. The build tooling is pretty much everyones universal pain point right now :/

[alexeagle]

No open issue, sorry.
Internally we have nice TypeScript and JavaScript rules for our build tool, http://bazel.io/
It has the advantage that we can easily hook up a file watcher and a background tsc worker to get immediate, incremental re-builds and execute tests and binaries. But we have not open-sourced all the bits yet and Angular needs something sooner.
Looking into Gulp 4 for better description of ordering constraints between tasks.
Will probably still use our custom Broccoli layer for incremental typescript compiles.
Also planning to incorporate docker as the environment spec for better portability and reproducibility.

[benlesh]

closing as inactive.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.