Webpacker: How to configure the generation of a server rendering output file?

[justin808]

Need

We need different env specific output configurations for server rendering.

Possible Solution

We can make different configuration options in webpacker.yml and the stub webpack config files that vary based on the Rails/Node Env (production vs. development vs. test).

For server rendering files, could we have some configuration of the entry file (or files) and where they get output, as well as differences in the webpack config?

So rather than having the webpack config named webpack/production.js, could we allow webpack/server/production.js or webpack/production.server.js?

For the webpacker.yml file, could we have a subsection called server under each env, like we have the dev_server section.

Like:

production:
  server:
   # These must be different than the defaults for client rendering. These files are not available 
   # in the view helpers. These are only available in the `Webpacker::Manifest#lookup` method,
   # specifying the option for the server bundle
   source_entry_path: packs
   public_output_path: packs

Server rendering code (for something like React on Rails):

Rails.root.join(File.join("public", Webpacker.manifest.lookup(bundle_name, server: true))).to_s

Note, the choice of an additional option server: true to lookup for server rendering. The option name "server" could be configurable. That being said, in many years of React on Rails, we came to the conclusion that only having ONE server rendering bundle makes sense, while for the client, we need to have lazily loaded small bundles.

Notes

• We can't have two watch processes writing to the same manifest.json.

[javan]

When you say "We" are you referring to React on Rails or is this a general purpose need?

could we allow webpack/server/production.js or webpack/production.server.js?

Webpacker doesn't mind. You can put any other files you want in there.

For the webpacker.yml file, could we have a subsection called server under each env

Can you give an example of settings that would be different in production vs. production.server?

And, generally, can you make the case for why these changes are needed in Webpacker? Could React on Rails manage its own configuration and build on top of Webpacker where needed?

[justin808]

@javan:

Questions

When you say "We" are you referring to React on Rails or is this a general purpose need?

any server rendering platform

Can you give an example of settings that would be different in production vs. production.server?

Compare ServerRouterApp.jsx to ClientRouterApp.jsx.

Those would be the entry points for server vs. client rendering.

Other differences are related to packaging, source maps, etc. for client (browser) JS execution rather than by MiniRacer's V8.

And, generally, can you make the case for why these changes are needed in Webpacker. Could React on Rails manage its own configuration and build on top of Webpacker where needed?

Sure, I can copy/paste the Webpacker code so that this happens correctly, in Utils.rb.

    def self.bundle_js_file_path(bundle_name)
      if using_webpacker?
        # Next line will throw if the file or manifest does not exist
        Rails.root.join(File.join("public", Webpacker.manifest.lookup(bundle_name))).to_s
      else
        # Default to the non-hashed name in the specified output directory, which, for legacy
        # React on Rails, this is the output directory picked up by the asset pipeline.
        File.join(ReactOnRails.configuration.generated_assets_dir, bundle_name)
      end
    end

That method is called by:

    def self.server_bundle_js_file_path
      # Either:
      # 1. Using same bundle for both server and client, so server bundle will be hashed in manifest
      # 2. Using a different bundle (different Webpack config), so file is not hashed, and
      #    bundle_js_path will throw.
      # 3. Not using webpacker, and bundle_js_path always returns

      # Note, server bundle should not be in the manifest
      # If using webpacker gem per https://github.com/rails/webpacker/issues/571
      return @server_bundle_path if @server_bundle_path && !Rails.env.development?

      bundle_name = ReactOnRails.configuration.server_bundle_js_file
      @server_bundle_path = begin
        bundle_js_file_path(bundle_name)
      rescue Webpacker::Manifest::MissingEntryError
        Rails.root.join(File.join(Webpacker.config.public_output_path, bundle_name)).to_s
      end
    end

Summary

1. I hate to duplicate/copy the rails/webpacker Ruby code to read the manifest.
2. This is 100% needed, at least by server apps that use React Router.
3. I'd like to configure the React on Rails generator to use the webpacker JS code.

[justin808]

@javan @gauravtiwari @dhh If we can solve this issue, I'll be able to ship React on Rails with very tight integration with rails/webpacker, including the use of the default webpacker install as part of the React on Rails default install.

Side question:

If the React on Rails install default install will use the webpacker default setup, should I have the React on Rails generator run the webpacker install if it has not yet been run? Or just error out?

This seems a bit analogous to how the installer for React requires that base installer to be run first.

[justin808]

A problem with the default configuration. We need to do something with regards to server rendering and CSS in the configuration. @javan?

This is the error:

Encountered error: "ReferenceError: self is not defined" 

The explanation is here:
https://github.com/shakacode/react_on_rails/issues/246#issuecomment-285610474

Just in case anyone will stumble upon this problem in the future:
My problem was, that I was using style-loader to include css into my react component, which embedded the styles into the DOM. This is not supported when using server side rendering, so I extracted the css to a separate file using extract-text-webpack-plugin.

That being said... watch mode works with the current setup. Just not the webpack-dev-server.

[justin808]

@gauravtiwari @javan @dhh Any thoughts on this one? This is the final issue from making the integration of https://github.com/shakacode/react_on_rails with Webpacker v3 💯 .

Reproduction steps are very simple:

First be sure to run rails -v and check that you are using Rails 5.1.3 or above. If you are using an older version of Rails, you'll need to install webpacker with React per the instructions here.

1. New Rails app: rails new my-app --webpack=react. cd into the directory.
2. Add beta gem version: gem 'react_on_rails', '~> 9.0.0.beta.12'
3. Run the generator: rails generate react_on_rails:install
4. Start the app: foreman start -f Procfile.dev
5. Visit http://localhost:3000/hello_world

Turn on HMR (Hot reloading)

1. Edit config/webpacker.yml and set hmr: true
2. Start the app: foreman start -f Procfile.dev-server
3. Visit http://localhost:3000/hello_world
4. Edit app/javascript/bundles/HelloWorld/components/HelloWorld.jsx, hit save, and see the screen update.

Turn on server rendering (does not work with hot reloading, yet, per Webpacker issue #732:

1. Edit app/views/hello_world/index.html.erb and set prerender to true.
2. Refresh the page.

This is the line where you turn server rendering on by setting prerender to true:

<%%= react_component("HelloWorld", props: @hello_world_props, prerender: false) %>

[justin808]

@dhh @gauravtiwari @javan is there anything I can do to help move this along? I've provided some very simple repro steps. I think having a separate server configuration option is key. This requires API changes, so I will not submit a PR unless we agree to the changes.

For right now, React on Rails supports the following by default:

• Use a custom webpack setup to create the server rendering bundle and don't hash it b/c it won't be in the manifest created by the client webpack setup. Note, I've heard there's a way that two webpack configs can append to the same manifest.

• Don't use the webpack dev server during development and React on Rails will check the manifest for a hashed name.

• Another keep in mind is that there should typically be ONE server bundle to handle the whole app (cached), but many client bundles to support quicker page loads.

[dhh]

We're not doing any server-side rendering, so I'm not up to date with what that entails, but since all the webpack configs supplied by Webpacker are simply defaults, why not just extend them with your react on rails installer for what you need?

[justin808]

Hi @dhh we need a different config setup per the bundle used for server rendering. React on Rails supports this via custom setups. Essentially, the 3 things that need to be slightly different:

• ENV: handled by webpack config files that match the ENV name
• Bundles: handled by having a bundle per file in the packs directory. One of these could be for server rendering, so this bundle includes all the entry points.
• Client vs. Server rendering in the webpack config. On the server side, all the CSS needs to be extracted from the JS code. There may be other differences as well.

[gauravtiwari]

Sorry didn’t get chance to try out server side rendering but will give it a try today and report back. As far as I understand each pack can be server rendered given there is no DOM related code.

[gauravtiwari]

Here is a simple setup to do server rendering with Webpacker using ExecJS (no webpack config changes required):

# app/helpers
module ApplicationHelper
  def react_component(name, props = {}, options = {}, &block)
    pack = Rails.root.join(File.join(Webpacker.config.public_path, Webpacker.manifest.lookup("#{name}.js"))).read
    renderer = ServerRender.new(code: pack)
    renderer.render(name.camelize, props)
  end
end

# app/views
<%= react_component 'hello_react', { name: 'World' }.to_json %>

# Adapted from react-rails
# app/lib/server_render.rb
class ServerRender
  # @return [ExecJS::Runtime::Context] The JS context for this renderer
  attr_reader :context

  def initialize(options={})
    js_code = options[:code] || raise("Pass `code:` option to instantiate a JS context!")
    @context = ExecJS.compile(GLOBAL_WRAPPER + js_code)
  end

  def render(component_name, props)
    js_executed_before = before_render(component_name, props)
    js_executed_after = after_render(component_name, props)
    js_main_section = main_render(component_name, props)
    html = render_from_parts(js_executed_before, js_main_section, js_executed_after)
  rescue ExecJS::ProgramError => err
    Rails.logger.debug err.message
  end

  # Hooks for inserting JS before/after rendering
  def before_render(component_name, props); ""; end
  def after_render(component_name, props); ""; end

  # Handle Node.js & other ExecJS contexts
  GLOBAL_WRAPPER = <<-JS
    var global = global || this;
    var self = self || this;
  JS

  private

  def render_from_parts(before, main, after)
    js_code = compose_js(before, main, after)
    @context.eval(js_code).html_safe
  end

  def main_render(component_name, props)
    "
    ReactDOMServer.renderToString(React.createElement(eval(#{component_name}), #{props}))
    "
  end

  def compose_js(before, main, after)
    <<-JS
      (function () {
        #{before}
        var result = #{main};
        #{after}
        return result;
      })()
    JS
  end
end

Finally the react component,

// Run this example by adding <%= javascript_pack_tag 'hello_react' %> to the head of your layout file,
// like app/views/layouts/application.html.erb. All it does is render <div>Hello React</div> at the bottom
// of the page.

import React from 'react'
import ReactDOMServer from 'react-dom/server'

const Hello = props => (
  <div>Hello {props.name}!</div>
)

global.HelloReact = Hello
global.React = React
global.ReactDOMServer = ReactDOMServer

export default Hello

[gauravtiwari]

This is another setup to do server rendering using hypernova:

// hypernova.js
const { join } = require('path')
const hypernova = require('hypernova/server')
const renderReact = require('hypernova-react').renderReact
const { environment } = require('@rails/webpacker')
const requireFromUrl = require('require-from-url/sync')
const detect = require('detect-port')

const config = environment.toWebpackConfig()
const devServerUrl = () => `http://${config.devServer.host}:${config.devServer.port}`

function camelize(text) {
  const separator = '_'
  const words = text.split(separator)
  let result = ''
  let i = 0

  while (i < words.length) {
    const word = words[i]
    const capitalizedWord = word.charAt(0).toUpperCase() + word.slice(1)
    result += capitalizedWord
    i += 1
  }

  return result
}

const detectPort = new Promise((resolve, reject) =>
    detect(config.devServer.port, (err, _port) => {
    if (err) {
      resolve(false)
    }

    if (config.devServer.port == _port) {
      resolve(false)
    } else {
      resolve(true)
    }
  })
)

hypernova({
  devMode: true,
  port: 3030,
  async getComponent(name) {
    const manifest = require(join(config.output.path, 'manifest.json'))
    const packName = manifest[`${name}.js`]
    const isDevServerRunning = await detectPort
    let bundle

    if (isDevServerRunning) {
      requireFromUrl(`${devServerUrl()}${packName}`)
    } else {
      require(join(config.output.path, '..', manifest[`${name}.js`]))
    }

    return renderReact(name, eval(camelize(name)))
  }
})

# config/initializers/hypernova.rb

require 'hypernova'
require 'hypernova/plugins/development_mode_plugin'

Hypernova.add_plugin!(DevelopmentModePlugin.new)

Hypernova.configure do |config|
  config.host = 'localhost'
  config.port = 3030
end

# app/views
<%= render_react_component('hello_react', name: 'World') %>

 # app/controllers
class PagesController < ApplicationController
  around_action :hypernova_render_support

  def index
  end
end

# Procfile
web: bundle exec rails s
watcher: ./bin/webpack --watch --colors --progress
hypernova: node hypernova.js
# webpacker: ./bin/webpack-dev-server --inline=false

Full guide here: https://github.com/airbnb/hypernova#rails

[gauravtiwari]

Perf Comparison:

ExecJS

user     system      total        real
0.010000   0.000000   0.010000 (  0.014621)
0.010000   0.000000   0.010000 (  0.011691)
0.010000   0.000000   0.010000 (  0.012252)
0.020000   0.000000   0.020000 (  0.018900)

Browser: 120-130ms

(tested on same machine running node server and rails server side by side )

Hypernova

user     system      total        real
0.000000   0.000000   0.000000 (  0.000698)
0.000000   0.000000   0.000000 (  0.001027)
0.000000   0.000000   0.000000 (  0.000531)
0.000000   0.000000   0.000000 (  0.000527)

Browser: 40-50ms

DISCLAIMER: Mileage may vary depending on your environment.

[gauravtiwari]

There is lot of gotchas involved though with server rendering like - HMR, inline styles, DOM related code won't work unless using isomorphic component but webpacker doesn't restrict server rendering in anyway. A pack can be rendered on the server if the component is isomorphic using any of the options above.

Encountered error: "ReferenceError: self is not defined"

Please turn off inline mode and dev server should work too: ./bin/webpack-dev-server --inline=false

As described above, server rendering doesn't require any config changes and it just works 👍

Closing this issue since server rendering is possible.

[justin808]

@gauravtiwari FYI -- in terms of performance, the number may not NOT the whole story given:

1. Server rendering involves a lot of data in a production app, like https://www.friendsandguests.com. There's a lot of time serializing and deserializing the data.
2. If running the Node Server on a different machine, you will have network latency

With any performance differences, it's worth considering why you're getting the differences.

Presented above, it looks like HyperNova is a slam dunk. However, my team built an express based node server for rendering with https://github.com/shakacode/react_on_rails and we had trouble overcoming the performance issues of the 2 points above.

[gauravtiwari]

@justin808 Thanks for sharing, added a disclaimer underneath the comment 👍 . May be, haven't tried this in production but I guess AirBNB uses HyperNova if I am not mistaken.

[mkalygin]

Sorry for posting in closed issue. I was looking for a solution to a similar problem. Maybe my investigation will be useful for others.

If you want to customize server pack config and client packs config, here is the trick I've came up with:

// config/webpack/environment.js
const { environment } = require('@rails/webpacker');

// Setup shared manifest for multi-compiler.
const ManifestPlugin = environment.plugins.get('Manifest');
ManifestPlugin.opts.seed = {};

// Convert environment to Webpack config.
const config = environment.toWebpackConfig();
const { entry } = config;

// Split entries for server pack and all the client packs.
// In my case server pack is called `server.js`.
const serverEntry = { server: entry.server };
const clientEntry = Object.assign({}, entry);

// Remove server entry from client entries.
delete clientEntry.server;

// Override default Webpack config for server and client.
const serverConfig = Object.assign({}, config, {
  entry: serverEntry,
  // your server pack config customizations...
});

const clientConfig = Object.assign({}, config, {
  entry: clientEntry,
  // your client packs config customizations...
});

// Use multi-compiler. Expose `toWebpackConfig` for external usage.
module.exports = {
  toWebpackConfig() {
    return [clientConfig, serverConfig];
  },
};