Products.cmfplone: PLIP: Resource registry improvements

[thet]

PLIP: Resource registry improvements #1955

Responsible Persons

Proposer: Johannes Raggam

Seconder:

URL: https://github.com/plone/Products.CMFPlone/issues/1955

There is a _Definitions_ section later in this PLIP, which explains some of the terms used here.

Abstract

• Get rid of the need of bundle resources.

• Instead, let _bundles_, which are technically just bunch of plone.registry settings, define their dependencies to resources directly instead via _bundle resources_.
  This makes the dependencies also extendible via just registry manipulation.

• Store the creation and modification dates of a registry entry.

• The feature of storing registry metadata must be implemented for plone.registry, as it lacks this feature.
  Via the Zope transaction history we can even get a diff between registry versions, if needed - but only for history aware ZODBs.

• If the modification date of the filesystem or plone.resource based _Resource_ (or any of it's dependencies) is older than the modification date of the bundle registry settings,
  then show a notification for the integrator to start a recompile of the bundle (through the web or via plone-compile-resources).

• If there is no compiled bundle or the RR is in development mode: then just include an automatically generated RequireJS require call to include all dependencies (which might be an option also in production mode with HTTP2).

• Add a button to delete the setting for a path to a compiled version of the bundle.

• Explicitly add an entry to show/delete a compiled plone.resource based version of the bundle.
  Those resources overload the ones in the filesystem.

• If recompiling TTW fails, automatically roll back.
  Offer a manual roll-back button anyways.

• Let RequireJS handle the JS dependency resolution and what to include in the compiled output via the settings of enabled, depends-on (dependencies to other bundles) and dependencies (dependencies to _Resources_).
  If this isn't possible, we can calculate all the _Stub Modules_ in Python code too.
  This makes the _Stub Modules_ obsolete.

Advantages

• No need for _Bundle Resources_ anymore - a concept which is very hard to explain.

• No need for _Meta Bundles_ anymore.
  We just include all bundles.

• Addon developers are encouraged to add their resources just to one of the standard bundles (plone or plone-logged-in).

• Addon developers can also create a new bundle and provide a compiled version of it.

• Integrators get notified when a bundle needs to be recompiled, because filesystem versions have changed.

• Integrators can better see if a resource or a bundle is shipped from filesystem or from plone.resource.

• Stub modules do not need to be defined anymore, and any JavaScript isn't included twice that way.

Motivation

Wipe away all the growing pains of our resource registry and the feeling to need to apologize for everything.

Assumptions and Risks

The PLIP https://github.com/plone/Products.CMFPlone/issues/1653 should be considered too.

• See how Webpack fits into that concept...

• Maybe RequireJS gets obsolete by Webpack? AFAIK they can well coexist. But I'm missing a bit of knowledge about the concepts and internals of Webpack.

• This still assumes to use LESS. But SASS seems to replace LESS everywhere.

• Maybe TTW compilation should be abandoned at all.

Definitions

• Resources: JavaScript or LESS filesystem resources.
  The have a name to refer to, in registry.xml defined as plone.resources/NAME.

  In case of JavaScript files, these are also RequireJS modules: JavaScript files, which make use of RequireJS define function to create a RequireJS module, associated with a name which can be imported with require.

  We make use of this with our Mockup Patterns [http://plone.github.io/mockup/dev/]. These are patternslib Patterns [http://patternslib.com/] with a JavaScript API in form of a object, which is returned by this module and gets initialized automatically when a class like pat-mymodule if found in the DOM.
  Also any JavaScript resource, which we depend upon. Be it a RequireJS module or not.

• Bundle resources: Special Resources, which actually define a bundle.

  It defines all the bundle's dependencies, initializes the patternslib registry and can run some other initialization code.

• Bundles: Bundles define dependencies to resources which are included in the rendered output.
  The have a name to refer to, in registry.xml defined as plone.bundles/NAME.
  The switch compile allows the creation of two types of bundles:

  • If compile is True: This makes it a RequireJS bundle, which should use the RequireJS define method to define the set of dependencies.
    This is then compiled with the command line script plone-compile-resources or Through the Web with r.js, the RequireJS compiler.

  • If compile is False: This makes Plone just output the JavaScript resources which are defined as _Resources_.
    It's also possible to just define one JavaScript file, which is then just output as is.

• Meta bundles: There are 3 meta groups: None, default, logged-in. Bundles of a _meta group_ (which is a property of Bundles) are concatenated in the HTML response.
  In case of None, they are not concatenated at all.

• Stub modules: List of resources, which are already included in another bundle and therefore are only included as "stub modules" without all their code in the compiled output of the bundle.
  he list of stub modules is an attribute of a bundle.

• Development Mode: If the resource registry is in development mode (this is a setting in plone.registry, in case someone needs to manually set it), then no _meta bundles_ are used and no concatenation or production mode caching URLs (++production++ ...) are used.

  • In development mode you can set a bundle to "Develop JavaScript" and "Develop CSS". In that case, the bundle will not use the compiled version but the resources - which the bundle depends on - directly.
  • In case of _Develop JavaScript_ and _compile is True_, RequireJS is used to load the resources.
  • In case of _Develop CSS_ and _compile is True_, the LESS JavaScript compiler is used to include and compile LESS files directly.

Proposal & Implementation

Probably a significant amount of work has to go into the implementation of this PLIP.
The proposal is outlined above.

Deliverables

...

Risks

See above.

Participants

...

Disclaimer

I didn't check against how settings are really named in the UI and in code. I just recalled them, so there might be some errors..

[hvelarde]

do you mind adding a definitions part to your proposal: what are _bundle resources_, _meta bundles_, _stub modules_ and why we need/needed them?

also:

• what's the difference between resource registry bundles in production and in development mode?
• why we need to recompile resources?

[thet]

@hvelarde @gforcada I have reworked everythign and included some definitions, made everything more clear and removed some errors.

@vangheem @bloodbare @robgietema @datakurre @ebrehault I would appreciate a review from you guys too :)
It sounds all good to me, but maybe I'm missing something.

@datakurre I don't know how webpack could play a role here.

@ebrehault This would make stub modules obsolete, more or less just for simplicity and because it's essentially not needed because people should extend one of the default bundles anyway. What do you think?

@pbauer That could maybe part of a discussion in Sorrento too (but I'm not there...).

/cc @jensens @rnixx

[ebrehault]

@thet sounds good to me :)
And I agree it might be quite a big job.

[datakurre]

@thet Webpack plays well with RequireJS. I don't believe that you need to think Webpack yet in this PLIP.

My webpack builds depend on config.js and less-variables.js -views, but config.js never included bundles and LESS cannot be removed yet, so I expect to be unaffected by these changes.

[hvelarde]

@thet keep it simple: don't mess around RequireJS now and don't try to guess anything. also, I still don't understand why we need to recompile resources.

IMO, Plone should ship with resources bundled; when a upgrade is done, a command line script managed by buildout could take care of the bundling of those resources only. any resources external to Plone (third party resources) must be just rendered as declared: if an add-on declares 3 JS, and 3 CSS, make no assumptions and just render that; all heuristics is broken.

is responsibility of each add-on developer to make thing play well; it's a decision of add-on developers if they want to use RequireJS or no; but Plone should not give support to add-ons not playing by the current rules.

what we need to have is a clear path to cook resources on add-on upgrades as we had before.

[djay]

@thet Does this proposal solve the problem of requiring a plugin reinstall to have updated filesystem compiled js/css be active? I can't see that explicitly stated but I might be missing it as a sublte implication.
This problem is a huge pain. You should be able to deploy a plugin and just have it work. Currently on a shared platform you have to go to every plone site and reinstall. If there are unrelated issues with reinstalling then you are in trouble, as your site is broken. I only want to reinstall a plugin if it has an actually upgrade step, not to deploy updated JS.

[hvelarde]

that's why I want to get rid of all the bundling process that happens on runtime; I insist that resources should be just registered and rendered without any need of interaction among them; I know that's far from optimal but this problem is really complex and we have no good solution available now.

just look at the insane process of updating any JS component in the core; the repos are continuously and the production cache of any site is destroyed every time a single line of code is changed. that's worst than what I was proposing, IMO.

please just make things easier.

[gp54321]

I noticed today that when uninstalling an add-on the resources can't be removed (there is no actual removing done in the exportimport directory modules). This seems well known since an addon like collective.glossary does not even include resource instructions in the uninstall profile. However when the uninstall is done, the plone-legacy bundle is not cooked, and the javascript of the removed add-on keeps loading in the browser. The resources can be deleted by hand in the resource registry but you have to be aware of the problem.

I wonder if a new version (possibly enhanced) of the addon is installed, the new javascript will replace the old in the plone-legacy. That would be very annoying if it is not the case.

Unless I am mistaken I don't see this requirement in your PLIP.
To me it looks more a missing feature than a bug, but I can file an issue if needed.

[thet]

Closing due to inactivity.