Webpacker: How to use this gem from within rails engines?

[fiedl]

I'm not sure if this is the right approach and maybe I've not understood this new part of the pipeline, yet, but:

Suppose, the major part of the app code lives inside a rails engine including all javascripts. And there are several main apps using that engine. The main apps only contain some layout changes and some minor patches.

Therefore, in order to integrate the webpacker gem into this setup, I'm trying to do the heavy lifting inside the engine, i.e. keep the work that has to be done inside all the main apps as little as possible.

How would I do that? Any pointers or suggestions are appreciated.

I'm not sure if this is a duplicate of https://github.com/rails/webpacker/issues/21. But in any case, I'd like to conclude this issue with a step-by-step guide how to approach this, for others facing the same use case.

What to do in the engine

1. Include webpacker in the *.gemspec file.
2. require 'webpacker' in the lib/foo/engine.rb.
3. ...

What to do in each main app

1. bundle install
2. Include ./bin/webpack-dev-server in the Procfile if using Foreman.
3. ...

[gauravtiwari]

@fiedl Out-of-the-box webpacker doesn't support engines yet. There are quite a few moving pieces that needs to be considered for this setup to work properly and usually people will have different use cases. I started doing some work on this but it's going to take time.

Feel free to leave any ideas/suggestions 👍

[fiedl]

@gauravtiwari Yes, after diving into this, I see your point. One would have to decide for each component where it belongs---to the engine, the main app, or both---and tie the pieces together in the right manner.

Some thoughts on this:

• The bin/webpack-dev-server is needed in the main app. One could add it to the Procfile of the main app to start the dev server with Foreman. Is there already a way to have the webpack dev server started by Pow?
• But one would need the ability to add a package from the engine.
• The app/javascript files have to be loaded from both, the engine and the main app. The main app files need to have the final say. I guess if there are several engines, the files from all engines need to be loaded.
• There are some files like .babelrc, .postcssrc.yml that would seem right in the main app root folder; but I'm not sure what these files are doing, yet.
• The config/webpack files appear to be tricky. In my use case, I would like to minimise the work to be done in the main app. But for other use cases, I imagine, the main app would need more control over the configuration. I really like how initializers work. I have most of them in the engine, but those of the main app have the last say in case of a conflict. Maybe we could utilise this mechanism: Would it make sense to tell webpacker in an initializer where to look for the webpack config? If no initializer is present, just use config/webpack from the rails root folder.
• For package.json and yarn.lock, one would need a mechanism similar to what is used with the *.gemspec, Gemfile and Gemfile.lock. If the engine decides that a certain package version is required, the package will be updated together with the engine when running bundle update my_engine from the main app. But the last say regarding the package versions has the main app as they are locked in the lock file. After all, the yarn.lock looks like a Gemfile.lock.
• This leads to the question how the upgrade process would look like from the view of the main app. I guess, just running bundle install from the main app would not cover any javascript package updates from the engine.

[himdel]

@gauravtiwari I started adding webpacker support to ManageIQ, where the UI is just an engine (so, app root != UI root), and we may need to support reading assets from multiple engines, and we need to output them to the rails root folder, not engine root.

To achieve that, I needed to override most of the webpacker methods to use the engine root instead of hardcoding Rails.root, I also needed to add a rake task that outputs Rails.root and call that from webpack config (didn't find another way of providing that path to webpack). (This also means that we need to include the initializer that overrides those paths from our Rakefile, which is a bit unfortunate :).)

.. And I still haven't figured out what would be needed to be able to call webpacker:compile, so far it it works if you manually call Webpacker.bootstrap but only when called from the root app, not when called from the engine.

I still haven't finished work on the "read assets from multiple engines" bit, but so far, I have a rake task that outputs a json of all the engines and their root paths (and filters those engines by existence of /app/javascript, until there's a proper way to register this), and intending to read that in the webpack config.

So.. I think at least these changes would be needed:

• don't assume all paths are relative to the current directory
• don't hardcode Rails.root anywhere
• come up with a sane way of passing those engine paths all the way to JS

If you're interested in the changes that we needed (and feel free to criticize and/or make suggestions), https://github.com/ManageIQ/manageiq-ui-classic/pull/1132 .

---

Furthermore (and these are only my personal opinion):

• we should default to expecting the assets compiled, UI devels can enable webpack-dev-server, and for the rest, it should Just Work as long as they don't make UI changes.
• we should never deal with yarn, yarn and webpack are separate tools, and webpacker's job starts when you have all the assets, not before
• all those safety checks (compile depends on verify_install, verify_install depends on check_node and check_yarn, ..) are out of scope and needlessly complicate these tasks. All we need is to see the real error when it fails.

EDIT: Oh, and if there was a nice way of disabling the assets:precompile hook, it would be lovely, when wrapping that task, you may want the other taks to run on precompile :).

EDIT2: Aand making all the compile-time yarn deps as --dev, so that non-dev is reserved for actual UI dependencies, but I guess that's a separate issue :)

[zacksiri]

Right now i'm doing this, but obviously the solution is quite incomplete. Here is what I'm doing, what I see as the potential solution

by default wepbacker will always look for the manifest in the /public/packs folder of the Root application, that's because the default webpacker.yml is configured as such. However when coming from an engine environment it should respect the setting of the engine's webpacker.yml as well. which means webpacker needs to be aware of multiple namespaces.

For webpacker to be successful in an engine environment we need things to be isolated like every other part.

Which means, when we install the engine into an app, it needs to have an 'install step' that basically installs the pre-built modules into the rails app /public/packs/engine/name/manifest.json

We will probably need to modify the stylesheet_pack_tag and javascript_pack_tag helper to be aware of namespaces. So basically if you want to use the engine's assets you should be able to do something like javascript_pack_tag 'engine/name/vendor' etc... or javascript_pack_tag 'vendor', module: 'Engine::Name' which resolves to /public/packs/engine/name/vendor.js

I will have more clarity on this as I develop the solution. right now for me developing the engine with JS code I am just using a symlink from the dummy app's public/packs folder to the engine's public/packs folder, which works for development purposes.

A question for @dhh is also that should the Root app using the engine have to worry about compiling an engine's assets? If not it should make thing simpler since we would expect the JS code from the engine to already be compiled and usable which would mean we just need to make the javascript_pack_tag and stylesheet_pack_tag aware of those assets based on namespaces. or the module: 'Engine::Name' option that gets passed in.

[ohadlevy]

I've made a POC using webpack and rails engines that I believe that can be reused to solve this request: see https://github.com/ohadlevy/foreman/commit/2afa796552c3331a1f22df7143ef6f9def733b67 for a reference.

[soundasleep]

I would love to see some progress on this feature - this is preventing me from using webpacker in my newest project.

[dhh]

Please Do Investigate 👍

On Jul 19, 2017, at 23:03, Jevon Wright [e-mail] wrote:

I would love to see some progress on this feature - this is preventing me from using webpacker in my newest project.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[ohadlevy]

we have been using webpack successfully (without webpacker) with rails engines on theforeman/foreman project.
they key was to create a simple ruby script that loads all engines paths, and pass that output as json to webpack, which in turn simply add those path to its lookup paths

[dhh]

Sounds like something we should be able to automate 👍

On Thu, Jul 20, 2017 at 7:01 AM, Ohad Levy [e-mail] wrote:

we have been using webpack successfully (without webpacker) with rails
engines on theforeman/foreman https://github.com/theforeman/foreman
project.
they key was to create a simple ruby script
https://github.com/theforeman/foreman/blob/develop/script/plugin_webpack_directories.rb
that loads all engines paths, and pass that output as json to webpack,
which in turn simply add those path to its lookup paths
https://github.com/theforeman/foreman/blob/develop/config/webpack.config.js#L30

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rails/webpacker/issues/348#issuecomment-316682127,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAKtcL8JHDO8sv-eFyRzPOOtBxPNcmoks5sP0GAgaJpZM4NUgTi
.

[mltsy]

Should the Root app using the engine have to worry about compiling an engine's assets?

On the surface, it's a fair argument that an asset engine (which is essentially what a webpack app gem is) could be expected to have all the assets pre-compiled, because... once it's released the assets aren't going to change... but looking at how assets in engines currently work, that is not the case, probably to allow including uncompiled assets within other uncompiled assets in the Root app (e.g. application.scss could use mixins from an engine's scss files) if that's the intent of the engine. I'm not totally sure whether that would ever be the case with a webpack engine I guess... is it feasible that there might be a use case for an engine that provides pieces to be used within a larger webpack app? Perhaps the engine would expose some Elm module or React component, and the Root app's UI would utilize that component from its own Elm/React app?

In Elm at least, that actually doesn't make sense as a use case, because all modules have to be explicitly included in the elm-package.json, and will then be added to the dependency graph independently of any Rails Engine. But I'm not sure the same can be said of every potential webpacked app - for instance, what if your Engine's app is just straight ES6 that exposes some modules? Could the Root app import those modules from the engine to be used as potential internal components of it's reactive UI?

If that kind of integration is possible (and useful, which seems likely to me if it's possible), then in order to facilitate it, a Root app obviously needs to be able to compile the engine's assets (after they are included and used however they are going to be used).

My main argument against allowing this type of integration, through a Rails Engine specifically, would be that it's mixing what are becoming ever more the separate responsibilities of the server-side app and the client-side app. That is, if your reactive UI requires some component, it should import it using a javascript dependency manager like npm, whereas the Gemfile should manage dependencies for your server-side Rails app (which may include some compiled reactive UI app to be used on the front-end, like any other asset to be sent to the client)

My question is: is that a good enough argument to make it impossible using webpacker (by assuming there will be no precompiled integration between the UI components of an engine and its root app, and hence not providing any considerations to facilitate such integration)?

[tomprats]

I was originally patiently waiting for this kind of support so I could upgrade a gem I manage to be used with webpacker, but @mltsy's argument resonated pretty well with me.

Javascript provided by an engine to be used through webpacker could just be put in (an) npm package(s). This would also allow the javascript to become part of the npm ecosystem, becoming a dependency or depending on other packages.

The engine could still provide CSS and JS when needed through sprockets. One downside is the potential overlap of an engine providing a page that depends on javascript that is also provided in its npm package. It could lead to a few use-cases of an end user having to download the same javascript code from multiple locations

[tomasc]

@mltsy @tomprats I partially agree with this view.

However, let's say the engine is a pluggable CMS that defines its own JS dependencies (similarly to gems in .gemspec). I would hope there would be a way for the main Rails app to pickup these dependencies as well.

As of now, in my case, most of the JS dependencies defined in a Rails engine are facilitated via https://rails-assets.org. It would be fantastic to be able to define them via webpacker.

[mltsy]

Yeah - this is a fairly unique situation where we have essentially two dependency managers in a single application (bundler and npm), and specifically the situation you're describing, where we have a Rails app with a bundler dependency on a Rails Engine, and an npm dependency on the "webpack app" (for lack of a better term, assuming it's not published as an npm package) that is defined in the Rails Engine. And that's the sticking point - the Rails engine "depends on" the webpack app that is implemented in its javascript/packs directory, but that dependency isn't really part of bundler or npm. So it's hard to say how/if it should be exposed in the contexts of npm and bundler. If we expose that somehow, we would essentially be adding javascript dependencies via the ruby dependency manager, which confuses things a bit when it comes to the javascript dependency manager... (there are some things included via an external channel, which creates somewhat of a mess when it comes to dependency management).

The best solution I can think of to handle that would be to somehow create/expose a kind of meta-package (or legitimate package?) out of the webpack app(s) contained within the Rails Engine, and automatically add that package to the packages.json of the Rails app before compiling with webpack (or maybe when the Engine is installed?). But this is all very theoretical, so maybe I should just shut my mouth and let someone who has attempted something like this share their wisdom ;)

[justin808]

@mltsy @tomasc @tomprats @ohadlevy @soundasleep @fiedl:

The beta version of React on Rails is built on top of Webpacker and here are the docs for using it.

I just released v9 beta.1, built on top of:

• https://github.com/rails/webpacker/pull/601
• https://github.com/shakacode/react_on_rails/pull/908

[chimame]

How about specifying more than one in source_path of webpacker.yml?

default: &default
  source_path:
    - app/javascript
    - engine/javascript
  source_entry_path: packs
  public_output_path: packs
  cache_path: tmp/cache/webpacker

Specify the entry point of the application and the entry point of the engine, and build it with the application.
If it is dynamic, it will be glad even if it can be done dynamically with ERB etc.

[tomasc]

Thanks @chimame, looks like a good idea, I will give it a try and report back.

[lazylester]

@chimame I have successfully done something similar to your suggestion. Since all the engines in my app are in vendor/gems, my source_path definition in webpacker.yml is:

source_path: '**/app/javascript'

And this pulls in the modules from all the engines as well as the main app.

This is working very well for me, _except_ I had to hack a single line in the rails webpacker npm module, as the manifest.json keys were not correct. I would happily create a PR for this, but I think there may be other use cases I'm not thinking of. Also, although my hack doesn't break the tests, I don't think this line of code is covered by tests and it's not clear to me why it was written that way.

I have a small example app with this implemented at https://github.com/lazylester/webpack_example

[mltsy]

Hmm... You're right! That's cool! At least partly... I mean that example app isn't quite the proof of concept that is necessary, because it doesn't actually require a module defined in the engine, but I bet it could!

I was slightly misunderstanding the role of npm and webpack. npm defines and downloads the sources (dependencies) from which webpack can choose what to include in any given pack. An engine is just another source for webpack to pull from, so it supplements the sources supplied by npm. Then webpack can choose from either npm's dependencies or the additional sources provided by the engine(s)...

Now... that is still a little bit disconcerting, just because those sources are outside of npm's dependency graph, meaning... the other thing we still need to solve is how to install the dependencies of the engine's pack(s). I see your "inventory" engine has no package.json (no JS dependencies) - that's not too likely in the real world, I imagine. Would you want to try adding and using one dependency and see if that works? I can't imagine running npm install on the main app would also install the engine's dependencies. But maybe there's a way to get around that... ? (I'm no expert in npm, so someone else here might have a better idea than I do about how to solve that issue)

[lazylester]

@mitsy you're right, I don't currently require any modules in the engine. I have done it elsewhere but forgot to include it in the example app. I'll update the example and post here. However my engines do not have any of their own npm dependencies, just local modules, within the engine, and a global (ractivejs) module that is in the main app's node modules.

So, yes, my approach has limitations. and doesn't cover many use cases. But it gets me to production!

[mltsy]

@justin808 - now that I understand what I'm looking for more I looked through the docs for using react_on_rails to see how you're handling this issue too. I see you're using a rails task to run a command provided by the engine (configuration.build_production_command) to compile the engine's pack, but (and this may be my ignorance about webpack) I see two issues in using this as a general solution:

1) It looks like that just runs webpack... when do the dependencies get installed? Does webpack install them? Does the user need to install them before running the task? (I see a yarn build:production version of the command in the test suite, but it looks like that script is not actually implemented or used)

2) If that command runs webpack with a totally separate webpack.yml, that wouldn't allow the parent application to include any of the engine's modules, etc. right? It's just compiling a totally independent pack? I see there's a comment just above the command saying if you don't want the engine to build the pack for you, that command can be set to nil, but I don't see any documentation on how to compile your pack in that case. Have you designed a workflow for compiling a webpack in the parent app that uses components/modules from your engine?

[mltsy]

Aha! Maybe we could use npm's local dependency feature?
https://docs.npmjs.com/files/package.json#local-paths

Yarn also respects this syntax... so we could tell the user to add this local dependency (file:vendor/gems/engine/? I'm not sure where the "package" root actually is - I would assume whatever directory the engine's package.json ends up in) to package.json in the parent app if they want to use any of the locally provided modules in their parent app's pack, and if not, provide a task for compiling the engine's pack independently! (could even make the installation part of an engine:install task)

I have to try this... when I get a chance... (if anybody else does, I'd love to hear the result)

[chimame]

@mltsy I think it's a good idea.

If both the idea to add to my webpacker.yml and the plan to add to yourpackage.json come true, you can do the following.

• Write javascript_pack_tag 'engine_ javascript' in View (if it exists in webpacker.yml)
• import engine from 'engine_ javascript' can be defined in JavaScript (if it exists in package.json)

[lazylester]

@mitsy I updated my example app https://github.com/lazylester/webpack_example.

The engine now has a local module of its own and npm modules. You'd probably have to build the packs for the engines in addition to the packs for the main app in a build script, but I assume that's not a big deal, since there has to be a build script anyway, right?

I'll generate a PR for the webpacker change that permitted this to work. It may or may not be accepted, depending on whether it breaks something else. As I mentioned before, the tests still pass.

[lazylester]

Pull request: https://github.com/rails/webpacker/pull/875

[mltsy]

Nice! So... I guess there are two use-cases for engines here which have different solutions:

1. I want to use an Engine to provide uncompiled modules for use in a parent app's pack(s)
2. I want to use an Engine to provide one or more compilable packs to a parent app

In case 2, that's what react_on_rails seems to be doing - it provides a rails task for compiling the engine's pack, and then you can use it like any other asset. @lazylester and @chimame's solutions are nicer ways to be able to do that without having to run a separate task, by including the engine's entry points in the webpacker sources of the main app. :star:

The use case I'm looking to solve for my personal use is case 1 though. And now that I think about it, we don't even need to include the engine's entry points in the webpacker sources to enable that, we just need to depend on it as a local dependency in package.json right? If that's true, it makes this all very clean, since there are orthogonal/independent solutions to each use case, and each can be employed depending on the purpose of the engine. (I still have to try the local dependency thing though for case 1, and the specifics of automating case 2 still need to be ironed out, since neither suggestion currently works with webpacker)

Since they seem to be orthogonal concerns, I suggest we open a new ticket for one of these use cases, and clarify the title of this ticket to represent the other - but I'm not sure which one corresponds best to the original post here... @fiedl ?

[chimame]

As a result of various thinking, I implemented it like this.

We are considering the following.

1. Packages managed by yarn can be managed on the Engine side
2. We assume that JavaScript on the Engine side is to be built in the upper application

1 means that it is not necessary to add the package on the upper side. This became possible by correspondence with @mltsy's comment.
There is also the idea that 2 was wanted to create on the Engine side while keeping the image of the JavaScript constructed by the upper application.

Please refer to this commit for details.

[etewiah]

My usecase is super simple. I just want webpacker to compile everything into one file in the standard rails javascripts location. In my usecase that will be:

property_web_scraper/app/assets/javascripts/property_web_scraper/my_webpack.js

I will then reference that js file as per normal from my application layout.

When I setup webpacker I was obliged to install it in the dummy app so the config file for example is here:

property_web_scraper/spec/dummy/config/webpacker.yml

I imagine I can set up a config to do this and invoke it somehow but I just don't know how :(

[tleish]

We are testing a solution for using webpacker within an engine using the engines dummy app and the asset pipeline. The general idea is:

1. Remove the webpack fingerprint hashing from generated files as it will be fingerprinted in the asset pipeline.
2. Export compiled packs to a namespaced folder inside the packs folder using the webpacker.yml (test/dummy/packs/my_namespace/file.js)
3. Include the test/dummy/packs folder within the engines asset pipeline (config.assets.paths << config.root.join('test/dummy/public/packs')
4. Include the packs file config.assets.precompile << %r{my_namespace/.*\.(?:js|js.map|css|png|jpg)\z}
5. Since the default webpacker precompile behavior is to automatically compile before after asset precompile, we disabled this default behavior ( ENV['WEBPACKER_PRECOMPILE'] = 'false' ), and instead have webpacker auto-compile with rake assets:precompile before the asset precompile is performed. That updated packs are automatically included with the assets.
6. Last, we monkeypatched the pack helper methods to use rails asset pipeline helper methods instead the engine was being used used by the parent application, but still use default pack helper methods when developing using the dummy app.

With the above changes, we can develop the engine in isolation using the standard webpacker functionality, but then in it's hosting parent, it can just use the generated assets when it performs rake assets:precompile. Monkeypatching the include pack helper will then allow us to use the pack include tags in the event that someday a solution is created for allowing parent and engines to both use webpacker, at which point the monkeypatch can be removed.

We made all the above configurable. For example, step 1 is controlled by fingerprint: true in the webpacker.yml file.

Questions:

1. Is there any reason we couldn't change step 5 to be the default behavior? The less we override changes to deviate from the main webpacker gem the better.
2. Is there any interest in including any of our changes as part of the main webpacker gem?

[gregogalante]

@dhh

What about a copy of source code as a node module on a temporary directory when assets are compiled?

Take this example:

• I have an engine with apackage.json file on root directory and assets on app/javascript/engine_name path.
• When i need to compile assets the webpacker:compile task takes all gem with webpacker dependency and a package.json file on the root (or with a configuration that tells rails to consider the gem as a webpack dependency) and copies them on a temporary directory (it copies only the package.json file and the app/javascript/engine_name directory).
• The webpacker resolved_pathsoption automatically contains also the temporary directory used.
• After the compilation of assets the temporary directory can be deleted.
• If I run rails server with RAILS_ENV=development the application should copy dependencies to the temporary directory on startup (so i can develop the application using the gem dependency as a normal npm dependency).

This method permits also to mantain the possibility to compile source code of the engine on app/assets/javascript (to be used with Sprockets) and to publish the engine javascript code to npm (similar to what is done on rails activestorage).

I have created a generator that should be used inside an engine to develop it with webpacker (https://ideonetwork.github.io/dakarai/#/pages/EngineGenerators). The generator should also update the dummy app to import engine assets and watch them on change.

[radeno]

This is quick hack when you use only one engine with webpacker. For example separated CMS UI from base rails.
Whole idea is about using symbolic files.

Just make Rails Engine with Webpacker (add into gemspec). All Webpacker required files will be in engine directory. Then symlink files or directories from engine directory to main directory:

• config/webpack
• config/webpacker.yml
• .babelrc
• .postcssrc.yml
• node_modules
• package.json
• yarn.lock
• public/packs

Javascript and styles are separated to engine.

[jrochkind]

I'm confused where this issue stands right now.

The webpacker README says "If you are adding Webpacker to an existing app that has most of the assets inside app/assets or inside an engine...", which sort of implies this is a supported use case. But the example following that suggestion actually isn't an example of "assets inside an engine" at all.

Is it actually a supported use case to have "assets inside an engine"? If so, can an example be provided? If not, should that suggestion be removed from the README?

[mltsy]

@jrochkind - It looks like the README is showing specifically how to include uncompiled assets from any directory (including an engine) - I haven't tried this, but it sounds like it may be possible just by adding the engine's path to the resolved_paths array. The trick is figuring out what that path is (if it's not part of your project's working tree). I would also like to see an example of what that directory reference should look like... if you figure it out, please post!

Actually here is an interesting example of someone including assets from a gem by vendoring the gem (unpacking it into /vendor): https://github.com/rails/webpacker/issues/57#issuecomment-330457616

From the rest of that discussion though, it sounds like the crowd is moving toward publishing a separate NPM package for any gem/engine that is supposed to work with webpacker...

[jrochkind]

I would suggest that if the README is going to mention engines, it probably should provide an example.

I do not vendor my gems into ./vendor, that isn't really a general purpose solution.

[inkhrt]

what exactly do you need help with?

On Sat, Apr 14, 2018 at 8:40 AM, Gregorio Galante [e-mail]
wrote:

What about a temporary copy of source code as a node module on a temporary
directory when assets are compiled?

Take this example:

• I have an engine with apackage.json file on root directory and
  assets on app/javascript/engine_name path.
• During development the dummy app package.json file define the engine
  dependency as a local dependency.

"dependencies": {
"engine_name": "file:../../"
}

• When i need to compile assets with the webpacker:compile it takes
  all gem with webpacker dependency and a package.jsonfile on the root
  and copy them on a tmp directory (it copy only the package.json file
  and the app/javascript/engine_name directory).
• The webpacker resolved_pathsoption automatically contains also the
  temporary directory used.
• After the compilation of assets the temporary directory can be
  deleted.

This method permits also to mantain the possibility to compile source code
of the engine on ``àpp/assets/javascript``` (to be used with Sprockets) and
to publish the engine javascript code to npm.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rails/webpacker/issues/348#issuecomment-381310855,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ah9VyP7bSPc-cgN7B5B1vpmGjoNGdheLks5toaf6gaJpZM4NUgTi
.

[jrochkind]

An engine is providing javascript assets.

In sprockets world, they are on the sprockets load path, and including them in the sprockets-compiled JS is as simple as require 'on/load/path/file.js' in your sprockets file.

In webpacker world, where the host app is using webpacker instead of sprockets (and the engine-provided js assets may be ES6 etc)... what does one do?

[gregogalante]

@inkhrt

In my company we have already created a generator used to create the described type of engine (https://ideonetwork.github.io/dakarai/#/pages/EngineGenerators?id=webpacker). The only thing that should be included inside the webpacker gem is a task that copies all gems javascript modules on the temporary directory when "rails assets:precompile" is executed.

The code used now is this:

# copy package.json to vendor directory
src = "#{MyEngine::Engine.root}/package.json"
dst = "#{Rails.root}/vendor/modules/my-engine/package.json"
FileUtils.mkdir_p(File.dirname(dst))
FileUtils.cp(src, dst)
# copy javascript directory to vendor directory
src = "#{MyEngine::Engine.root}/app/javascript/my-engine"
dst = "#{Rails.root}/vendor/modules/my-engine/app/javascript/my-engine"
FileUtils.mkdir_p(File.dirname(dst))
FileUtils.cp_r(src, dst)

It should be executed automatically on assets compilation and we need to find a way used by gems to tells webpacker that they have javascript dependencies that needs to be compiled (the first idea was to consider all gems with a package.json file on the root path).

[jejacks0n]

I would like to add to this a bit. Maybe it's been discussed already, but what I've seen is just pathing issues when used inside a dummy app or in a non-straightforward way. I've written a fairly widely used (maybe less so these days) javascript testing library for rails and so understand some of the nuances of javascripts and engines.

I'm doing this in a non-open source engine for what it's worth, but the same thing applies if I were to upgrade my teaspoon project to use webpacker. Ok, so typical setup, an engine, inside the project is a spec/dummy app that rails more or less generates when generating a new engine project. There are also tests that are sitting next to the dummy app, and that use the dummy app to utilize the engine in a fully functioning rails app for testing behaviors etc.

Now, worth mentioning that this dummy app is in fact fully functional at this point, everything actually works when you cd into spec/dummy and run it in the development environment utilizing the webpack-dev-server. Everything also works when you fire it up in the test environment (e.g. rails s -e test) and the assets are compiled into public/packs-test as the default configuration dictates.

However, when running the tests, which are run from the engine root and not the dummy app root, webpacker will fail to compile when capybara fires up the dummy app, and it appears the reason is because pathing is assumed to be based on where you're running a thing from, and not where your rails application lives. I tend to believe this is a bug.

I tracked it down and I've monkey patched the compiler class to resolve this issue, and if anybody cares to make a PR that would be awesome -- I'm mostly sharing this for the dev team to help identify potentially how better to integrate with existing rails engine core concepts because it's a fairly simple adjustment to gain a lot of potential benefit. The thing to note here is that I'm only chdiring into rails root to run the command found at compiler.rb#L59, and when I do so everything begins to work as it should.

require "webpacker/compiler"
class Webpacker::Compiler
  private

    def run_webpack
      logger.info "Compiling…"

      stdout, sterr, status = Dir.chdir(Rails.root) do
        Open3.capture3(webpack_env, "#{RbConfig.ruby} ./bin/webpack")
      end

      if status.success?
        logger.info "Compiled all packs in #{config.public_output_path}"
      else
        logger.error "Compilation failed:\n#{sterr}\n#{stdout}"
      end

      status.success?
    end
end

I guess I should add that I'm not trying to load javascripts or assets from my engine directly, that's done through npm or yarn and is simply bundled in the same library as the gem by specifying a package.json -- anyhow, this is more dealing with how to test your implementation in an engine. If anybody wants to pick my brain on this more I'd be open to walking someone through the whole structure that I've setup and what I think is working nicely as a distribution platform for an engine with complex logic and javascript assets.

[markets]

Hi 👋 #1836 has been recently (~ 8 hours ago) merged, hope this helps...

Specially this file: https://github.com/rails/webpacker/blob/master/docs/engines.md

[mltsy]

Awesome! Thanks @markets
I'm no longer developing with webpacks in an engine currently, so I'm not in a good place to evaluate this, but just to help resolve the conversation (and possibly move toward closing this ticket), could you identify whether this PR solves one or both of these use cases discussed in this ticket (mentioned in a previous post above):

1. I want to use an Engine to provide uncompiled modules for use in a parent app's pack(s)
2. I want to use an Engine to provide one or more compilable packs to a parent app

[adamwgriffin]

I tried following the docs that @markets linked to at https://github.com/rails/webpacker/blob/master/docs/engines.md and I'm having a really hard time getting it to work. Has anyone else tried following these docs? If so, do you have a repo with a working example that I could look at?

[jejacks0n]

It’s in an early stage, but the concepts are in place at:

https://github.com/legworkstudio/protosite/tree/master/spec/dummy

I’m not at a machine, otherwise I’d provide more insight. Note the node_modules symlink, and the place I patch webpacker:

https://github.com/legworkstudio/protosite/blob/master/spec/dummy/config/application.rb

The patch in https://github.com/legworkstudio/protosite/blob/master/spec/dummy/config/webpack/compiler_patch.rb is what I suggest as a fix here but haven’t been bothered to submit a PR until I hear more feedback for core devs.

On Jan 9, 2019, at 5:49 PM, Adam Griffin [e-mail] wrote:

I tried following the docs that @markets linked to at https://github.com/rails/webpacker/blob/master/docs/engines.md and I'm having a really hard time getting it to work. Has anyone else tried following these docs? If so, do you have a repo with a working example that I could look at?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[adamwgriffin]

Thanks @jejacks0n

[jejacks0n]

@adamwgriffin it's not exactly what you were looking for, because I did it without having a walkthrough to follow, so I'm pretty sure it doesn't follow that exactly.

[adamwgriffin]

@jejacks0n it's not exactly what I was looking for but maybe this will work better 👍

[ghaydarov]

Has anyone else tried following these docs? If so, do you have a repo with a working example that I could look at?

@adamwgriffin I was able to get it working on a sample app below.

https://github.com/ghaydarov/webpacker-with-engines

[adamwgriffin]

Thanks @ghaydarov that's just what I was looking for!

[emanuel-campos]

@ghaydarov

Webpacker can't find c1-application.js in <root_path>/webpacker_engine_app/public/packs/u-packs/manifest.json. Possible causes:
1. You want to set webpacker.yml value of compile to true for your environment
   unless you are using the `webpack -w` or the webpack-dev-server.
2. webpack has not yet re-run to reflect updates.
3. You have misconfigured Webpacker's config/webpacker.yml file.
4. Your webpack configuration is not creating a manifest.
Your manifest contains:
{
}

[zedtux]

Thank you @ghaydarov very much for the nice repo, but can you give some instructions about how to redo the same please?

In my engine's repo I don't have the webpacker:install task, therefore when I do app:webpacker:install:react it's complaining :

webpack binstubs not found.
Have you run rails webpacker:install ?
Make sure the bin directory or binstubs are not included in .gitignore
Exiting!

[zedtux]

I got Webpacker working well within my Rails engine following instructions from https://github.com/rails/webpacker/blob/master/docs/engines.md and then fixing few issues.

The gem is here : https://gitlab.com/pharmony/active_record_migration_ui

[angarc]

@zedtux

Quick question, I'm a little new to webpacker and rails 6 in general. I followed your guide, and now I'm just wondering what the next step is? Would I run bundle exec rails app:webpacker:install? When I run that, I get another error about not knowing how to build a rake task app:template. I appreciate you putting up the step by step guide, I'm just not too sure what to do afterward.

Update
I figured out what the problem was. I was supposed to run the command from the test/dummy/ directory, and not the directory of the rails engine. However, no my only question is what exactly I should put on my <%= javascript_pack_tag 'application' %> because when just putting application doesn't work. I saw that output path for the packs was in the public folder as specified by your guide. However, I can't seem to point my javascript_pack_tag there. Should it be something like <%= javascript_pack_tag 'public/my-engine-packs/application' %>? Sorry if this is a stupid question.

Another update
Okay I see where I went wrong... sort of. I moved my javascript folder to the app directory like any other normal rails 6 application. I want my rails engine to have bootstrap on it. I've already installed it using webpacker successfully. Now, the only problem is that the bootstrap styling is not displaying. I just get the regular default browser styling. When I checkout my logs I get this:

ActionController::RoutingError (No route matches [GET] "/my-engine-packs/js/application-0c695baf97e03e4d45ba.js"):

I remember setting my-engine-packs as the public_output_path in my webpacker.yml. Do you know what I could go about fixing this?

[zedtux]

@angarc yeah webpacker installation within an engine is not strait forward.

When I run that, I get another error about not knowing how to build a rake task app:template.

As you said in your first update, this is actually updating the test/dummy app. What I did here was to look into the source code of webpacker and see what was supposed to do the app:template and I did it manually but basically it was adding the webpacker* stuff to the config/ folder and the package.json file (from my memory).

I just copy/past those files from the test/dummy to the gem root and then started to use webpack as in a normal Rails app so putting the javascript_pack_tag in my gem's layout file.

Aside node : @gauravtiwari the webpacker generator should be updated so that it behaves differently when running within a gem.

When I checkout my logs I get this:

ActionController::RoutingError (No route matches [GET] "/my-engine-packs/js/application-0c695baf97e03e4d45ba.js"):

If you followed what is written here : https://github.com/rails/webpacker/blob/master/docs/engines.md you should then have in your engine file :

    initializer "webpacker.proxy" do |app|
        insert_middleware = begin
                            MyEngine.webpacker.config.dev_server.present?
                          rescue
                            nil
                          end
        next unless insert_middleware

        app.middleware.insert_before(
          0, Webpacker::DevServerProxy, # "Webpacker::DevServerProxy" if Rails version < 5
          ssl_verify_none: true,
          webpacker: MyEngine.webpacker
        )
      end

but also :

config.middleware.use(
  "Rack::Static",
  urls: ["/my-engine-packs"], root: "my_engine/public"
)

The latter one is the trick that will catch the requests to the /my-engine-packs URL, and serves the file from your gem's public/my-engine-packs/ folder.

Aside node : @gauravtiwari the webpacker generator should also amend the gem's engine file so that it adds those configs.

[angarc]

@zedtux
Is "Rack::Static" a dependency? I get this error: undefined method 'new' for "Rack::Static":String (NoMethodError)

And yeah, I tried following the steps provided but the javascript_pack_tag isn't working. At least, not when I do this: <%= javascript_link_tag 'application' %>. What's interesting though is that when I change it to something like <%= javascript_link_tag 'my_engine/application' %>, webpacker throws an error about it not being able to find it. So, I'm assuming that it is indeed finding that javascript file, however, I don't see any bootstrap styles on the page, so I don't know. Maybe I didn't install bootstrap correctly. Is there another way I can test this to see for sure whether or not the javascript_link_tag is finding the application? I don't think it is because I'm getting the error:

ActionController::RoutingError (No route matches [GET] "/my-engine-packs/js/application-0c695baf97e03e4d45ba.js"):,

However, I know that webpacker works because I was able to successfully install bootstrap and jquery using yarn and webpacker, and for that I can't thank you enough.

Update
I tried console logging in my application.js file and I got this error in my developer console.

The script from “http://localhost:3000/my-engine-packs/js/application-1749718b34b4b77f3945.js” was loaded even though its MIME type (“text/html”) is not a valid JavaScript MIME type.

Do you think this might be the problem?? Or at least a major part of the problem?

Another update
Well I don't know if this is correct, but changing "Rack::Static" to Rack::Static (I.e., I'm passing a constant rather than a string) didn't throw an error. Nonetheless, I'm still getting this error:

ActionController::RoutingError (No route matches [GET] "/my-engine-packs/js/application-0c695baf97e03e4d45ba.js"):

Also, I probably forgot to mention that I'm trying to do this with rails 6.

[zedtux]

I don't know if this is correct, but changing "Rack::Static" to Rack::Static

Yeah sorry, I forgot about this, the doc is wrong, it's clearly Rack::Static 👍. Maybe you could open a PR in order to change the doc?

The script from “http://localhost:3000/my-engine-packs/js/application-1749718b34b4b77f3945.js” was loaded even though its MIME type (“text/html”) is not a valid JavaScript MIME type.

Yeah this is the reason why it isn't loading the JS, so the CSS.

This part is covered by the README.md :

Note: You need to allow webpack-dev-server host as an allowed origin for connect-src if you are running your application in a restrict CSP environment (like Rails 5.2+). This can be done in Rails 5.2+ in the CSP initializer config/initializers/content_security_policy.rb with a snippet like this:

 Rails.application.config.content_security_policy do |policy|
   policy.connect_src :self, :https, 'http://localhost:3035', 'ws://localhost:3035' if Rails.env.development?
 end

Don't forget to restart your server when you do so.

Also do you have the bin/webpack and bin/webpack-dev-server files?

Also, I probably forgot to mention that I'm trying to do this with rails 6.

I'm using Rails 5, but based on the changelog files I saw as of now, there is no breaking changes on Rails/webpacker, so shouldn't be a problem.

[angarc]

@zedtux

Okay, so I added that line to the content security configuration, but I'm still getting the same mime type error. And yes :) I copied the bin/webpack and bin/webpack-dev-server from a regular rails 6 application.

I see that both bin files reference a Gemfile instead of a gemspec, I don't know if that's a problem.

[zedtux]

Oh, thinking of your error once again and I realised I was wrong, this error is because the request to JS is returning HTML (look in the browser’s console, from the Network tab, and click the js file line, you’ll see that the preview shows HTML code.)

Look at the HTML, it should be a Rails error page, so that you see what’s the error is.

As far as I remember, this error was about that Rack::Static didn’t find your pack file.
I remember debugging this from the Rails console, in order to define the right path to be passed to Rack::Static.
Have a look in my gem’s engine file see what I passed as arguments.

Regarding your second point, you should have a Gemfile in your gem, don’t you?

[angarc]

@zedtux Your gem serves static assets right? I see that there are views in your gem. I cloned your gem from Git lab, updated the rails gem to rails 6, and tried putting some views that I wanted while keeping your exact configuration—I didn't change anything. Interestingly enough, I still got the same error:

The script from “http://localhost:3000/my-engine-packs/js/application-1749718b34b4b77f3945.js” was loaded even though its MIME type (“text/html”) is not a valid JavaScript MIME type.

ActionController::RoutingError (No route matches [GET] "/my-engine-packs/js/application-0c695baf97e03e4d45ba.js"):

Maybe there is something that I'm just doing wrong. I setup a github repo of a Rails 6 Engine that I created following the rails engine guide step by step. Here is the link:

https://github.com/angarc/my-engine

Again, thank you for responding. I know you're most likely busy but it means a lot that you're still trying to help! :)

And yes I do have a gemfile!! sorry about that, I had forgotten that the gemfile just references the gemspec.

[zedtux]

Can you please describe how you've tested my gem please ?

The philosophy of my gem is "webpacker is not required to use it" so that means webpacker helps me to develop (HMR) and then build a bundle to be distributed within the gem so that the end user _just_ have to load the JS and CSS, but doesn't need webpack.
This is the reason why I replaced the javascript_pack_tag and stylesheet_pack_tag helpers by the very simple JS script in the app/views/layouts/active_record_migration_ui/application.html.erb file loading from the manifest.json file the CSS and the JS.

Testing the gem archive

You have to start the Docker container and then run the rails build which produces a gem file I'm then installing manually using gem install --local <path/to/the/gemfile.gem>.
See the Releasing a new version section of the README.md file.

Developing the gem (with hot reloading)

To develop the gem (you change the gem's sources and you see immediately the result in the host Rails app) :

1. In the Rails app hosting the gem, add the gem to the Gemfile with a :path pointing to the gem's sources
2. Add an initialiser defining the webpacker_host to the URL of the running gem's webpacker server (In my case localhost:3036) so that it (re)loads from the gem's webpacker.
   ruby ActiveRecordMigrationUi.configure do |config| config.webpacker_host = 'localhost:3036' end

The latter is the trick to get the JavaScript loaded from the gem's webpacker.

---

I cloned your gem from Git lab, updated the rails gem to rails 6, and tried putting some views that I wanted while keeping your exact configuration [...]

I would suggest you to first do the same things without changing the Rails version.
When you get a working version, then upgrade to Rails 6 and test again.

Actually this is interesting me as your test could reveal that I would have to rework my gem for Rails 6, so I'll carefully look at your test results.

I still got the same error:

The script from “http://localhost:3000/my-engine-packs/js/application-1749718b34b4b77f3945.js” was loaded even though its MIME type (“text/html”) is not a valid JavaScript MIME type.

I'm a bit confused here as the URL is /my-engine-packs while with my gem that path is /ar-migration-ui-packs.
Did you copy/past from a previous comment, you are you actually really getting this error?

Again, thank you for responding. I know you're most likely busy but it means a lot that you're still trying to help! :)

You are very welcome, I know too much how hard it is to be alone facing a big issue like this one. You can count on me to help as much as I can ! 👍

https://github.com/angarc/my-engine

Yeah that's a good idea, it will be easier for me to help you !

Please first review this comment and let me know the output of this. Then I'll check your repo.

[zedtux]

Awesome, congrats @angarc 👍

[vyavahare-kishor]

Hello, @angarc

Could you please update you repo (https://github.com/angarc/my-engine), so I can take help from it, as I also have to add a engine app within my root app.

Thanks in Advance.

[zedtux]

@vyavahare-kishor you can also check mine at https://gitlab.com/pharmony/active_record_migration_ui

[n4rq5]

So after 20+ months the Webpacker, which should replace old assets pipeline eventually, has not stable way to support engines development.
The official guide (https://github.com/rails/webpacker/blob/master/docs/engines.md) is not working for me and it is very "hacky" in nature to put it mildly.
Are there any plans to support this use-case at all in any way or is this issue considered effectively dead now?

[gregogalante]

@n4rq5 If you want i have a pull request with some updates on the documentation (https://github.com/gregogalante/webpacker/blob/patch-2/docs/engines.md). Based on the read documentation there are two more things to do:

• add webpacker as a dependency of your engine
• install webpacker inside the test/dummy app (at the end of the second step)

If you need more help i have a project with a working implementation of webpacker inside an engine.

[tvdeyen]

The great thing about sprockets (the Rails asset pipeline) is that it "just works". Having to follow outdated documentation to implement the same code in all engines over and over again is far from ideal.

Webpacker - as part of the Rails ecosystem - should also "just work" for engines. Rails supports engines as a first class citizen, so should webpacker. Internally every Rails app is even only just an engine.

That said, I know that there are several use cases of engines providing assets nowadays:

1. Engines just providing assets for the main app. Remember rails-assets.org?
2. Engines having its own bundle of assets that should be compiled along with the apps assets

The first type of rails engines will vanish with the raise of npm and webpacker. Users can - and will most likely already - use the npm package instead. We do not need to support this with webpacker IMO.

The second one though is vital for this community. There are tons of CMS, admin panels, shop systems, user forums and alike that are all Rails engines providing their own bundle of assets, that ideally should "just work" with webpacker as well.

For those engines one key benefit of sprockets is its virtual filesystem. Like with Rails classes and views you can require a file no matter if it's inside the main app or an engine. You can even place a file in the main app and Sprockets/Rails will pick this up instead. The engines one is overwritten.

All of this is not possible with Webpacker today.

It would be great if all the engine authors that have a working webpacker installation with the above mentioned features would send PRs to update webpacker itself, instead of "just" updating the documentation.

Also it would be great to get some "official statement" from the rails core team on this. Without that it is difficult for contributors to help.

Thanks for webpacker! It works great in apps. Lets make it happen for engines as well.

[n4rq5]

@tvdeyen is absolutely right.

Sprockets handled all the heavy-lifting tasks out of the box and saved authors a lot of headaches.
I was able to require gem/engines assets everywhere without a hassle and it always worked.

Right now if when I am writing engine I have no out-of the box and stable documented way to develop and provide front-end assets to the app consuming the engine. I am not sure if this comes from lack of interes from the community or simply everyone lost it hope and moved on leaving idea to use webpacker behind.

Documented process with copying binstubs, configuring webacker.proxy etc. does not seam to serve it's purpose in a long run. It's a temporary workaround, not a solution in my opinion.

It's possible that the best solution is to encapsulate assets in separate NPM package. That's fair and reasonable but I believe rails core team should address it and define it as "recommended" practice with at least basic generators/installers support to start with.

Rails was always opinionated in many aspects but when it comes to using Webpacker flow in Engines it surprisingly lacks any suggestions how to approach it at all and this is not encouraging for anyone to flip the switch and forget about sprockets.

[jrochkind]

I agree Engine use is painful, and its unfortunate.

Having looked into it, I couldn't come up with any great solution. (A future version of webpack (not webpacker, webpack itself) that supports plugins may help, but I'm not sure the ETA for that).

The pain is kind of a necessary consequence of swithing to webpack and it's better integration with non-Rails-specific JS, which was really necessary and the right move or at any rate is a done deal.

I think delivering an engine's assets as an NPM package is pretty much the only solution. This does leave some outstanding pain points though:

• You might want to make sure to always release an npm package at the same time as you release the rubygem, with the same version. Some automated tooling to make this happen would be great, that is easy to use for people not already familiar with npm release process. (I believe Rails does this and may have automated tooling for it, but pretty heavyweight rails-specific custom tooling).

• If you are releasing an npm package with the same version number as a rubygem, you are intending them only to be used together. If using the engine rubygem with version X, you are expected to be using the npm package only with version X too -- any other mix of versions might not work as expected. This sticks to the semantics of sprockets-based engines, where a given version just has it's own JS assets, you don't try to mix and match JS from one version with another (allowing such would be a significantly added maintenance burder, for no real gain).

  • I believe Rails itself has this expectation too for Rails npm packages like actioncable etc. But I believe even Rails has no way of enforcing this. If you upgrade your Rails version but forget to upgrade your (eg) rails-actioncable npm package, you will get no warning or error, but it might not work and is not a good idea. I can't figure out any great way to solve this.

I think solving those things (and documenting as a recommended pattern) would go a huge way to making it clear how to release a Rails engine that has JS assets, using webpacker, with similar desirable semantics and no significant additional pain, to the sprockets-world engines.

But it's pretty unclear how to accomplish these things. I agree that the currently documented path is... weird, and not a good idea, I don't think it's actually going to work well or reliably, by it's very nature it's not really doing things compatible with how webpack works and I believe is going to cause problems (with how deployments work among other things).

[n4rq5]

I believe the problem with aligning versions of engine and NPM package can be avoided if the NPM package is not published as a separate entity but is included in the engine codebase and installed by yarn locally from engines directory.

The only semi clean development workflow i found was creating private NPM package in subdirectory of the engine and, installing webpacker in "dummy" app and install the package in the app using yarn link hook.

This passes responsibility of compiling assets to main app webpacker and webpack entirely and I believe it's most versatile approach as it allows the app to import engine assets at will, tree-shake them or simply create it's one pack to include them as w whole.

There are rough edges to smooth out and I believe most of them should be solved on webpacker gem level by better support of the use case.
Most problematic thing is actually exposing the engine package files to application webpack itself. I think webpacker should expose some mechanism for engines initializers or installer scripts to easily tap into webpack file system resolver mechanism behind the curtain. yarn link works for me locally while developing but making it work in production and CI/CD flow would be huge PITA.

[jrochkind]

installed by yarn locally from engines directory

I was unable to figure out any way to get that to happen, in a way that would work with standard deployment scenarios. Because the relative and absolute paths of a gem dependency can change between local machine and remote deployment machine and the relative path on remote deployment machine can depend on exactly what ruby technologies you are using (bundler local or not, rvm or not set up how, etc).

If a future version of yarn supports custom plugins, so there could be some custom dynamic logic, that could do it. But without that, yarn (outside of rails or webpacker's control) wants a path as a simple string in the package.json/yarn.lock -- and we want to be able to commit those files to the repo, and not have them have to change depending on deployment environment.

I couldn't figure out a way to do it with yarn.link either, without having to predict a fixed unchanging deployment environment as above, which is not how we want things to work -- as you mention. I wasn't able to come up with even any conceptual workaround that could be implemented just at the webpacker level, without changes to yarn and/or webpack themselves.

[dhh]

Maybe @wycats has an idea here, given his work on yarn?
On Fri, Jan 24, 2020 at 1:23 PM "github.com" [e-mail] wrote:
““installed by yarn locally from engines directory””

“I was unable to figure out any way to get that to happen, in a way that would work with standard deployment scenarios. Because the relative and absolute paths of a gem dependency can change between local machine and remote deployment machine and the relative path on remote deployment machine can depend on exactly what ruby technologies you are using (bundler local or not, rvm or not set up how, etc).

If a future version of yarn supports custom plugins, so there could be some custom dynamic logic, that could do it. But without that, yarn (outside of rails or webpacker's control) wants a path as a simple string in the package.json/yarn.lock -- and we want to be able to commit those files to the repo, and not have them have to change depending on deployment environment.”

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[tvdeyen]

Hi @dhh thanks for paying attention :pray:
What are your thoughts on this topic in general? Should webpacker support engines out of the box or should engine authors figure it out on their own?

[dhh]

I’d very much like to see engines supported by webpacker. 

On Fri, Jan 24, 2020 at 1:43 PM "github.com" [e-mail] wrote:
“Hi @dhh thanks for paying attention 🙏
What are your thoughts on this topic in general? Should webpacker support engines out of the box or should engine authors figure it out on their own?”

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[tvdeyen]

I’d very much like to see engines supported by webpacker.

@dhh Yes! Thank you so much :heart: (in many ways)

[jejacks0n]

The way I've handled this somewhat recently is to create a npm package and a gem with the engine. This could be a bit of a bummer in some cases, but may provide some projects with the best scenario to be honest. I came to like the decoupling of the backend logic from the front end logic, and that it may create a better ecosystem for variation.

There was a patch required for webpacker to understand a path (linked near the end). All that being said, here's a link to a project (small, incomplete though, incomplete code) where I took this approach: https://github.com/jejacks0n/protosite#development--contributing

I link directly to the how I've setup the development environment with bundler and yarn using yarn link to reference the local package.

The notable and relevant bits that might be helpful if someone decides to take this approach would be the gemspec and node package, and how we can use the gemspec in the Gemfile:
https://github.com/jejacks0n/protosite/blob/master/protosite.gemspec
https://github.com/jejacks0n/protosite/blob/master/package.json
https://github.com/jejacks0n/protosite/blob/master/Gemfile#L6

And the dummy app, notable here is where we have to patch some things for webpack/er:
https://github.com/jejacks0n/protosite/blob/master/spec/dummy/config/application.rb#L12
https://github.com/jejacks0n/protosite/blob/master/spec/dummy/config/webpack/compiler_patch.rb#L8 (I think I had to make this understand paths, but it's been a while and may not be entirely relevant anymore).

That last bit is the only thing out of the ordinary that was required to make it all work -- otherwise it was pretty similar to how you might setup an independent gem and npm package and working on it was pretty nice too.

[fiedl]

There is no way to have something like a package.json.erb, right?

// main_app/package.json.erb
{
  "name": "main_app_with_webpacker",
  "dependencies": {
    "my_engine": '<%= MyEngine::Engine.root %>',
  }
}

This would allow to have the engine's dependencies specified in #{MyEngine::Engine.root}/package.json and import the engine's module without publishing the engines npm package to a separate npm package repository. The npm package would just live inside the gem.

[jrochkind]

Right, you can't do that, a package.json.erb. Because the package.json is interpreted by Javascript toolchain utilities that don't know about ERB, would need to be enhanced to know about ERB and execute it in the ruby Rails app context where the MyEngine constant is available -- and probably not particularly welcome such a patch, because they are not Rails, webpacker, or even ruby-specific tools. As far as I can figure out, it's not a change webpacker alone could introduce, it would need to be the JS toolchain -- specifically, I think, yarn (which interprets the package.json in a webpacker setup, rather than letting npm itself do it).

I believe it would need to be yarn, which is why I'm interested in yarn's announced interest in a future version supporting plugins. We could imagine a webpacker/rails-specific plugin. But I don't know the ETA for such a thing in yarn.

[fiedl]

I don't want this to get hacky, but just as a thought experiment: Executing shell commands inline inside the package.json is also not supported by yarn install, right?

Otherwise, something like this would open a door to injecting the engine path:

// main_app/package.json

{
  "name": "main_app_with_webpacker",
  "dependencies": {
    "my_engine": '$(ruby -e "require \"./config/application\"; print MyEngine::Engine.root")',
  }
}

[jrochkind]

I doubt it, why don't you try it and let us know if you find an implementation?

[AfolabiOlaoluwa]

https://github.com/rails/webpacker/issues/348#issuecomment-578138091

I honestly will very much like to see engines supported by webpacker out of the box @dhh. It surely will make engines more modular or self-reliant. It's really sleek.

[andrewhaines]

Here's how I got this working, loading Stimulus controllers from within a gem / engine and having them compile for Stimulus within the main Rails app.

# main_app/packages.json
{
  "dependencies": {
    "@rails/webpacker": "^3.2.1",
    "my_gem": "../../gems/my_gem", # Or you can point to a GitHub repo, etc.
    ...
  }
}

// main_app/application.js

import { Application } from "stimulus"
import { definitionsFromContext } from "stimulus/webpack-helpers"

// The usual stuff
const application = Application.start()
const context = require.context("./controllers", true, /\.js$/)
application.load(definitionsFromContext(context))

// Pointed at my gem
const gem_context = require.context('my_gem/app/javascript/packs/controllers', true, /\.js$/)
application.load(definitionsFromContext(gem_context))

Note: I had some trouble trying to figure out how to register these. Using the following made it easy to see what was registered when:

console.log("controller definitions:", definitionsFromContext(context))
console.log("controller definitions:", definitionsFromContext(gem_context))

Then in the gem:

# package.json
{
  "name": "my_gem",
  "version": "0.0.1",
  "description": "testing...",
  ...
}

Finally, I created my Stimulus controller(s) as usual:

import { Controller } from "stimulus"
export default class extends Controller {
  connect() {
    console.log("It's working...")
  }
  ...
}

Seems to do what I need. I'm happy for any feedback as to whether I've missed something, though..

[thebravoman]

I kind of got it working with @andrewhaines approach, but it worked only if I know the path to the engine. Which works only if the engine is in the host_app/gems/ folder, but this does not work when the engine path to the filesystem is unknown

[jrochkind]

The problem of course is that the relative/absolute path of an engine in the file system will often differ between development and production, or possibly on different systems; the bundler/rubygems system is not meant for this to be fixed and unchanging from system to system.

[thebravoman]

I kind of got my answers - I don't think I really like them, but I guess I would have to accept them.

https://discuss.rubyonrails.org/t/webpacker-presents-a-more-difficult-oob-experience-for-js-sprinkles-than-sprockets-did/75345/13

https://discuss.rubyonrails.org/t/how-to-pack-js-from-a-different-gem-engine-in-rails-6-webpacker-it-seems-frustrating/75364/8

My conclusion for the moment seems to be - if you have a non trivial .js feature you'd need a npm package and get the dependencies in a "webpackER/webpack" standard way.

So I would "just" split my engines in two - leave the controllers in the engine, move the js to npm packages, maintain two separate stacks, build procedures and versioning. No irony or sarcasm here, just listing what would probably be involved if others come to this conclusion also.

[jrochkind]

Yep. I think separate npm package is going to remain the only reasonable solution. (POSSIBLY changing if yarn plugins become a real thing).

I think we could really use tooling to make managing this easier.

Even Rails doesn't actually use separate versioning. There are npm packages for eg actioncable, activestorage, and even ujs-rails that are always released with same version numbers as the corresponding (and same-named) rails gems. If I understand right, every single ruby gem release gets a corresponding npm release with the same version number -- even if there have been no changes to the JS code in the npm package. The intent is that an app will always use the npm packages with the identical version to the ruby gem in use. And of course, all the code lives in the same repo for Rails.

One problem is that I'm not sure this is intent is well-documented, and there aren't (to my knowlege) any tools available to the developers of the app using these dependencies to make sure the npm and gem dependency versions are kept in lockstep sync. I suspect there are a lot of apps out there using mismatched versions -- which works, unless/until it someday doesn't, and when it doesn't it could hypothetically be very confusing to debug what's going on if you don't think to consider the issue. What if there were a tool that would ensure that your app had sync'd versions of the npm packages and the rubygems, and errored or warned you if it didn't? (ideally at bundle install and/or yarn install time, less ideally at app boot time, less ideally of all at JS use time).

I believe Rails does have tools on the side of Rails publishing itself to always and automatically publish the npm package(s) whenever rails ruby gems are published. But this tooling is tied into Rails custom code, it isn't easily available to someone else who wants to do similar. What if there were a tool that added on to the bundler-provided rake release task to make sure a corresponding npm package in the same repo had the correct lockstep sync'd version (a bit tricky because npm versioning uses slightly different 'syntax' then rubygems, especially when it comes to prereleases), and were automatically released?

I think with the right tooling, there'd be much minimized pain of "splitting engines in two". But as it is, there is significant increased challenges and more importantly scope for easy to make (and routinely made) mistakes, both to dependency maintainers and dependency users.

It might be more profitable to focus on such tooling to reduce the pain of a dependency split between a rubygem and an npm package, rather than working out what seems like inevitably hacky and unreliable methods of avoiding the split.

[thebravoman]

Thanks @jrochkind. I had this internal problem with versions many times, where the same "feature" is separated in different projects that have different versioning. My example is with a "feature" in the platform separated in three stacks - js front end+rails server+python data processing. I came to the conclusion that I don't need to keep them in sync. It's nice to have, but all I needed was a well maintained changelog and a couple of tests to make sure each version works. We later found out we are doing a completely different release cycles for the three parts and syncing them only on a major release which was working like a charm.

That being said I would of course be interested to use any tooling that connects npm and ruby closer. What I was missing, but kind of understood from this tread and a few others is sprockets is there to stay, but webpacker is the way to go as there as npm packages for actioncable, activestorage, rails-ujs and probably others.

[tvdeyen]

Yes, I also believe separate NPM packages are the way to go for engines.

I just switched Alchemy today from the hack mentioned in the engines.md file to a separate NPM module and some Rails installer magic. This is a very clean approach that I am very happy with. I think other engines will follow.

Just have a look into that diff https://github.com/AlchemyCMS/alchemy_cms/pull/1853/files

🎉

[jrochkind]

@tvdeyen are you choosing to release npm packages with same versions as your ruby gems, with every release, as rails does? If so, do you have any tooling to make that harder to mess up?

Do you have anything in place to ensure the user is using an appropriate or compatible version of the npm gem, corresponding to the ruby gem version they are using? Or if not "ensure", to even let them figure out what versions are compatible?

These are the two areas I could see leading to problems, and could really use some 'best practices' or examples or recommendations on.

[tvdeyen]

@jrochkind No, not yet. But I could imagine some kind of .alchemy-version.yml file (You will actually remember the famous spree_version.yml file aren't you 😆?

This is something that could probably live in Rails::Engine and be provided by Rails

@dhh how do you manage this in Rails gems and NPM modules? Is there something that could be make accessible for Engine authors? Would be great.

[tvdeyen]

@jrochkind something like this? 🤔

Edit: Version that uses the actual resolved npm package version

initializer "alchemy.check_package_version" do
  yarn_list = `yarn list --silent --flat --depth 0 --pattern alchemy_cms`
  package_version = yarn_list.match(/\d\.\d\.\d/).to_s
  if Gem::Version.new(package_version) < Alchemy.gem_version 
    abort "Your Alchemy npm package is outdated, please run `yarn upgrade @alchemy_cms/admin`"
  end
end

There are still a couple of caveats here:

1. It assumes you use a triple digit version number (The regex can easily be adopted to use 2 or 4 digits)
2. It is pretty slow to run the yarn list command in a ruby process. This adds a couple of milliseconds (~1000ms on my machine) to your app start up time. Maybe we only make this check in Rails.env.development?
3. Assumes your npm package version string is compatible with Gem::Version
4. Assumes that the released npm package has always the same version of the engine. Maybe the gem author holds a list of compatible npm package versions in a separate list and compares against it?

WDYT?

[jrochkind]

Yeah, I think this approach makes sense, but I suspect there will be all sorts of edge case details to get right, it'll have to be worked out. I believe for pre-releases, different version strings are required.

I think it would be nice if there were an out of the box maintained solution to this, perhaps from webpacker or rails. I think it's currently these issues are currently the biggest additional pain to separating JS into an npm package; with them solved it could seem no more challenging than the old sprockets-focused solution of packaging your JS in the gem.

[valterkraemer]

This is not perfect, but works pretty well.

It figures out what the paths are to the engines on the file system, and then adds them as entry points to the parent Webpacker.

Get paths to engines on file system

#! /usr/bin/env ruby
# file: get_engine_paths

require 'bundler'
require 'json'

# gem names in Gemfile
engine_names = ['my-gem-name']

engine_paths = Bundler.load.specs
  .select{ |dep| engine_names.include?(dep.name) }
  .map{ |dep| dep.to_spec.full_gem_path }

puts engine_paths.to_json

Needs to be executable (chmod +x get_engine_paths)

Make Webpacker find packs in engines

// file: webpack/environment.js
const { environment } = require("@rails/webpacker");

const { execSync } = require("child_process");
const { basename, resolve } = require("path");
const { readdirSync } = require("fs");

const babelLoader = environment.loaders.get("babel");

// Get paths to all engines' folders
const scriptPath = resolve(__dirname, "./get_engine_paths");
const buffer = execSync(scriptPath);
const enginePaths = JSON.parse(buffer);

enginePaths.forEach((path) => {
  const packsFolderPath = `${path}/app/javascript/packs`;
  const entryFiles = readdirSync(packsFolderPath);

  entryFiles.forEach((file) => {
    // File name without .js
    const name = basename(file, ".js");
    const entryPath = `${packsFolderPath}/${file}`;

    environment.entry.set(name, entryPath);
  });

  // Otherwise babel won't transpile the file
  babelLoader.include.push(`${path}/app/javascript`);
});

module.exports = environment;

Run yarn install in engines when yarn install is run in parent

Add to package.json "postinstall": "./engines_yarn_install.js"

#!/usr/bin/env node
// file: engines_yarn_install.js

const { spawn, execSync } = require("child_process");
const { resolve } = require("path");

// Get paths to all engines
const buffer = execSync(
  resolve(__dirname, "./get_engine_paths")
);
const enginePaths = JSON.parse(buffer);

enginePaths.forEach(enginePath => {
  spawn("yarn", ["install"], {
    env: process.env,
    cwd: enginePath,
    stdio: "inherit"
  });
});

[jrochkind]

@valterkraemer does that solution assume the path is always the same in every environment, or does it actually look it up "live"? At asset compile time? Asset delivery time?

Of course the path is usually diferent between a dev and production deploy, and sometimes can be different between different production deploys.

[valterkraemer]

It looks it up at asset build time, so it should have no problem with changing paths.

[tubbo]

@valterkraemer I have a similar approach, but it uses bundle info on the CLI to get a particular gem's information rather than relying on a pre-built JSON. It works pretty well on my own test app but I haven't actually done anything with it:

https://github.com/rails/webpacker/compare/master...tubbo:add-installed-engines-to-additional-load-paths

You use it by calling the gem() function with the name of your gem, and adding to the module resolve paths in env config:

const { environment, gem } = require("@rails/webpacker")

environment.config.resolve.modules = [ gem("my-engine") ]

module.exports = environment

Then in your JS code:

import MyEngine from "my-engine"

This does require your code to be well namespaced, there's no automatic namespacing...it's just as if the files lived side by side in your app/javascript folder. So files in different gems with the same name would conflict with one another, and I suppose whichever file Webpack found first would be the one it used. But I think that's a decent trade-off for now.

[jrochkind]

@tubbo that's exciting. Do you think it would be feasible to turn it into a shareable package so other people can use this technique with share code? I guess it's functionality spans both JS and ruby, so maybe not?

[valterkraemer]

Thanks @tubbo. Weren't aware of bundle show, can now use that to get rid of the get_engine_paths file!

[brendon]

Not sure if this'll help anyone but I had good success with the following. In my setup I have a folder components within the root of my Rails application that then contains a bunch of engines.

1. In your engine, create a directory: app/javascripts/engine_name (e.g. components/blogs/app/javascripts/blogs
2. Within that directory you can do what you want. In my case I had a controllers directory. Within that I had the typical Stimulus.js loader (index.js):

// Load all the controllers within this directory and all subdirectories.
// Controller files must be named *_controller.js.

import { Application } from "stimulus"
import { definitionsFromContext } from "stimulus/webpack-helpers"

const application = Application.start()
const context = require.context(".", true, /_controller\.js$/)
application.load(definitionsFromContext(context))

A slight difference in this one is the require.context(".") instead of require.context("controllers") as otherwise this will look at the main app's controllers directory.

1. In config/webpacker.yml add a resolved_paths (or additional_paths if you use a newer version of webpacker).

resolved_paths: ['components/blogs/app/javascripts']

1. Now in your main app you can import your engine's controllers like so:

import 'controllers' // Should already be here
import 'blogs/controllers'

1. It's important to namespace your engine's javascript folder so that things don't collide.

Hopefully it's easy enough to see how this could be expanded for uses other than Stimulus.js.