Ghost: No more Grunt!

[ErisDS]

As part of the push towards Ghost 1.0.0, and getting rid of things which get in our way, we're going to look towards getting rid of Grunt. The tools we have now were first created before we split the admin client out of the main repository. The current versions are designed to match how the old tooling worked, but with the new setup.

They work but are slow, hard to use and confusing. Some tools are for installing, some tools are for developing and some tools are for releasing. The tooling isn't really good at one particular job.

The goals for the new tooling are:

1. An amazing set of developer tools that make it really easy for people to contribute and for us to do our jobs.
2. A very separate set of "install" tools. From Ghost 1.0, installing is going to be handled by the Ghost CLI tool. We're going to be recommending this for all production installs, and discouraging installing from git for production.
3. A single command for the purpose of converting source code to a ready-to-run build.
4. A release script which does ALL THE WORK and makes releasing super-crazy easy.

Installing from git will be for those people developing Ghost, so the tools we have in the repository should be geared towards making it super easy to develop.

In Ghost-Admin, I don't think we need much on top of ember-cli itself.

• [ ] what to do about the buildAboutPage task?
• [ ] can we run csscomb without Grunt/Gulp?

In Ghost, I want to remove the release-specific tools, and focus on having an amazing set of tools for contributors. I recommend that we use Gulp rather than Grunt, as it's much faster, and setup any npm aliases that are common amongst dependencies (like npm test).

• [x] New dev task that supports both client & server watch & livereload
• [x] Tools for watching just client or server?
• [x] Setup tool, for initialising the different repos on a particular PR/commit?
• [ ] Test tools
• [ ] Coverage tools
• [x] Linting tools

Aside from the development tools, which will do watching & live reloading, we should have one single command which will ensure the state of the source code (sync the submodules), do all build tasks to convert source code into built code, and also provide an opportunity for any minification or optimisations needed for production. I think this will likely be a gulp build command, aliased to npm run build.

• [ ] gulp build task

All release-specific code (handling npm dependencies, version bumping, generating contributor lists (except for the ones needed for the about page), zipping, etc will live in a separate place (although will probably also be built with Gulp).

• [ ] remove release-related code

[aileen]

I think in addition to csscomb we would need linting for the client as well? Or is there a way to implement that within ember-cli? 🤔

[kevinansfield]

ember-cli can run jshint/jscs as part of the test suite, maybe we want to move in that direction? Only downside would be that Travis wouldn't give an immediate indication that it's the code style causing problems rather than the functionality but on the plus side builds should be quicker as the whole container setup doesn't need to be done twice

refs:

• jshint: https://github.com/ember-cli/ember-cli-jshint
• jscs: https://github.com/kellyselden/broccoli-jscs

Maybe worth swapping jshint+jscs for eslint? https://github.com/ember-cli/ember-cli-eslint (Note: may be blocked on this https://github.com/DockYard/ember-suave/issues/113)

[kevinansfield]

WRT csscomb, I think having it as part of the gulp tooling is probably the best first step, then we can look at making a broccoli plugin that runs it as part of the ember-cli build process later on.

[aileen]

We had a bigger discussion about what we're going to include in our tasks and how they shall be named. Here's a quick summarise of the result:

As described in the issue, Ghost-admin will use ember-cli commands. No need to use grunt or gulp here anymore.
As an exception, we could include an interims task to do the csscomb on the admin until there is a better solution, like a broccoli plugin.

Here the planned tasks on the server side:

• gulp setup ghost-ref admin-ref
  The initial setup task to install dependencies on client _and_ server side. By default, gulp setup should use master for both and updates the submodule to the commit ref from master. _But_, we want to make it possible to work with different branches or PRs. E. g.:
  gulp setup pr/1234 pr/5678 will checkout pr/1234 for ghost and pr/5678 for admin, as well as installing the dependencies. It should also work with branch-names.
  An additional option --force will do a dependency fresh install (there will be an alias for this as well: gulp ffs 😉)
• gulp dev
  The task to work in development mode. This will start the ghost server as well as the client watch. Filechanges in server _and_ client will be watched and emit a livereload.
• gulp server
  The task, when you're only focused on the backend and don't care about the client. Will start the server and livereloads on any changes on server-side.

[kevinansfield]

By default, gulp setup should use master for both and updates the submodule to the commit ref from master.

I think it needs to pull in Ghost-Admin if the submodule doesn't exist yet (in which case it would be master) but after that unless ghost/ghost-admin refs are passed in it leaves the submodules/refs alone and runs the install commands for whichever branches you already have checked out. That way if branches are checked out manually you can still run gulp setup to run all the npm/bower install tasks.

If you ever want to force a reset to master you can run gulp setup master master?

[aileen]

Thanks! That's what I wanted to say, yeah 😛

Using master master instead of --force sounds like a good plan. And in that case we wouldn't even need to check for a third option in our arguments.

Cool!

[kevinansfield]

Umm, --force would still be necessary, we discussed that it would remove existing node_modules and bower_components dirs before doing the install.

When I said force a reset to master I meant switch everything back to the master branches as in your writeup you mentioned that gulp setup would default to master but I suggested it should default to whichever branches are currently checked out.

[aileen]

Oops... Alright. Gotcha! Sorry for the confusion.

[aileen]

@ErisDS:

Setup tool, for initialising the different repos on a particular PR/commit?

At the current state, I made it possible to checkout to different (existing) branches and PRs. Shall I implement a solution to initialise certain commits as well?

[ErisDS]

I don't think it's necessary to support commits right now, if we find we need it, we can add it later 👍

[kirrg001]

When ember-cli is not installed,i get the following message.
Error: spawn ember ENOENT.

Is it possible to show "npm install -g ember-cli"?

[kirrg001]

Windows:
gulp dev always shows spawn ember ENOENT.
But when i navigate to core/client and run ember build manually, it works.

[kirrg001]

Tasks which are left.
@AileenCGN Could you please confirm in DM?Thanks!

• [ ] finish gulp testing tools https://github.com/AileenCGN/Ghost/commit/d0d4f4472f76e437b8a19dc177b5281825569946
• [ ] add built task
• [ ] add coverage task
• [ ] finally remove grunt
• [ ] test on windows

[ErisDS]

I don't think we benefit from the speedups that gulp provides, because we need to do so much in sequence.

Also because we're working around the natural way that gulp works, we seem to have a lot more code, which is a lot more fragile & I think most of us who work on Ghost regularly are falling back to doing steps manually rather than trying to fix bugs we encounter in the gulp tooling.

E.g. submodule updates are not working on a fresh clone, it always says "no need to update".
Also e.g. gulp dev often doesn't seem to rebuild the admin panel - I think it takes longer than it takes to start the server but it's not at all clear from the output

The gulp setup -a=master -g=master command is awesome when it works, but I think it would be possible to achieve something similar with grunt?

We've also not touched test tooling yet.

I think a call needs to be made as to whether to pursue this or revert. I'm not familiar enough with gulp really to know if we can do better.

[kirrg001]

I never used gulp before and i can't tell what is the nature of gulp and what not.

They work but are slow, hard to use and confusing

This was raised as initial statement/problem with our existing grunt tooling.

In my opinion the new gulp commands (e.g. lint, deps, setup) are easy to use and not confusing. And i also don't think they are slow, in my opinion it's only the ember build command, which is soooooo slow and it's widely known in the community. And that's why gulp dev behaves a little annoying.

Even though, there is a good reason to revert. See why:

• test commands are missing
• built command is missing
• optimise all other commands
• testing on windows is missing

All these things are eating lot's of time, which i think we don't have right now. ✌🏻 ⏲ ⌛️
That's why i would move the gulpfile into a branch, finish later if we want and ensure all grunt commands are working as expected.

@kevinansfield
Was the ember build process always so slow? Did you see the ember build perf guide? (I am pretty sure you did)

[acburdine]

I forgot to respond to this, but I've used gulp in a number of other projects and have a decent idea as to how it works.

I've been following gulp development, and although development on it is a bit slow, gulp 4 will come with a lot of benefits that I think ghost could use. The primary one is that sequential tasks will be supported out of the box with gulp 4, without a need for a sequential plugin like run-sequence. This should serve to reduce the amount of code that we need, as well as its fragility.

Another optimization we could use is a library called execa. I've been using it in the CLI project, and it provides a lot of benefits on top of child_process, first of which is localized path variables. This means that we can have it use local deps over global ones, which should fix a decent amount of the code in that area.

I also think that the primary speed benefits that we'd see are likely not implemented yet, namely the release and test processes. I'd have to look at if this is possible, but we might (with gulp) be able to run the ember build commands in parallel, meaning we could shorten the release time by a decent amount. We could also potentially run the various test groups in parallel.

That being said, I do agree with @kirrg001 that it's probably better at this point to optimize our grunt tooling and move gulp into a branch. Once gulp 4 is finally released we could re-look at this.

[kevinansfield]

Was the ember build process always so slow? Did you see the ember build perf guide? (I am pretty sure you did)

It's been slower 😉 but yes, building from scratch will always be pretty slow as there is a huge amount of transpilation and sourcemapping happening, with no existing build caches the first build takes ~1min on my machine. If I cancel gulp dev and start it again the ember-cli build caches are used and I see initial build times of ~15sec but I rarely stop/restart gulp dev when focusing on the client side. When only touching server-side files the ember build time after the very first build shouldn't matter, the built files are already present and served by ghost immediately thanks to gulp running the processes in parallel.

The majority of ember-cli's perf focus (and the mentioned guide) has been on improving the rebuild times as that's generally the most painful area - after changing a client file you want to be able to see the results quickly and for Ghost-Admin this is actually really good, most rebuilds are ~1sec. Recent ember-cli versions have included optimisations for the number of files created/used during building so that should definitely be noticeable for initial builds if disk access is a bottleneck, I think there is still quite a bit of on-going work in that area to bring the initial builds down.

[kevinansfield]

If we're switching back to the grunt tooling I don't think there's any point bringing the client test tasks back - they limited the access to the built-in ember test command which meant I never used them and frequently pointed people to ember test rather than any grunt command. With the separated client/server tests it also made little sense to run both together as changes on one side would never affect the other.

We could definitely improve the documentation around ember tests, if we can point to the core/client dir and say ember test is usable from there then ember devs will already know what to do but we can also provide some information for non-ember devs showing commands for filtering specific tests and running in "server mode" so that the tests you care about re-run any time a client file is changed.

[ErisDS]

So the decision here is to switch back to the old grunt tooling, put the gulp tooling on a branch and try again when gulp 4 is out and it might work better for us.

We're NOT going to bring back any Grunt tooling that was specific to Ghost-Admin and not really used, in favour of documenting how to use the ember tooling.

The only tool we had in gulp that we do not have in Grunt is gulp setup -g=commitish -a=commitish which we can attempt to port to grunt if there is time. It is a useful tool, but not as required as the test, coverage & build tools that we are missing.

We can also review the gruntfile to see if there are simplifications that can be made, especially relating to the now much more independent Ghost-Admin.

[kirrg001]

Do we want to leave this open or label with later?

[ErisDS]

Labeled with later & closing for now.