Products.cmfplone: Plone should work when auto include is disabled

[mauritsvanrees]

Plone 5.2, Python 3.

When you have this in your zcml:

<meta:provides feature="disable-autoinclude" />

then z3c.autoinclude does not get loaded. But when you do this, the instance does not start:

Traceback (most recent call last):
  File "/Users/maurits/tmp/ploneenv/lib/python3.7/site-packages/zope/configuration/xmlconfig.py", line 391, in endElementNS
    self.context.end()
  File "/Users/maurits/tmp/ploneenv/lib/python3.7/site-packages/zope/configuration/config.py", line 703, in end
    self.stack.pop().finish()
  File "/Users/maurits/tmp/ploneenv/lib/python3.7/site-packages/zope/configuration/config.py", line 868, in finish
    actions = self.handler(context, **args)
  File "/Users/maurits/tmp/ploneenv/lib/python3.7/site-packages/z3c/autoinclude/zcml.py", line 50, in includeDependenciesDirective
    dist = distributionForPackage(package)
  File "/Users/maurits/tmp/ploneenv/lib/python3.7/site-packages/z3c/autoinclude/utils.py", line 70, in distributionForPackage
    return distributionForDottedName(package_dottedname)
  File "/Users/maurits/tmp/ploneenv/lib/python3.7/site-packages/z3c/autoinclude/utils.py", line 94, in distributionForDottedName
    raise LookupError("No distributions found for package `%s`; are you sure it is importable?" % package_dottedname)
LookupError: No distributions found for package `plone.app.contenttypes`; are you sure it is importable?

When I try to work around this, I simply get into the next problem, and another one, etcetera. Problem above seems to be that the package plone.app.contenttypes has includeDependencies in its zcml. And this does not work when z3c.autoinclude is disabled.

Background: I tried to pip install Plone, and that gives problems with z3c.autoinclude, so I tried to disable this. See also https://github.com/zopefoundation/z3c.autoinclude/pull/2 and https://www.fourdigits.nl/blog/installing-plone-5-using-pip/

What seems to be needed, is (please check the box when the PR is merged):

• [x] Remove includeDependencies from plone.app.contenttypes.
• [x] Remove includeDependencies from plone.app.iterate.
• [x] Remove includeDependencies from plone.app.linkintegrity.
• [x] Remove includeDependencies from plone.restapi.
• [x] In CMFPlone or possibly elsewhere: <include package="plone.resource" file="meta.zcml" /> See comment below.

And as extra, maybe we could make it easier in plone.recipe.zope2instance to set the disable-autoinclude feature.

Does this sound like a useful plan?

[mauritsvanrees]

I am preparing pull requests for this. But which zcml should we load. You can go through the list of dependencies in setup.py and check for each one if there is a configure.zcml or meta.zcml, but that is a bit tedious.

I have found that the easiest way to see which ZCML is really needed, is adding some print statements in z3c.autoinclude/dependency.py before includableInfo returns a result:

if self.context.project_name == "plone.app.iterate":
    print("")
    from pprint import pprint
    pprint(result)
    print("")

Or simply this, to catch them all:

logger.warning("Package {0} is using includeDependencies "
    "or includeDependenciesOverrides!"
    .format(self.context.project_name))
from pprint import pprint
pprint(result)

For plone.app.iterate it printed this:

{'configure.zcml': ['plone.locking',
                    'plone.memoize',
                    'Products.CMFCore',
                    'Products.CMFEditions',
                    'Products.CMFPlacefulWorkflow',
                    'Products.DCWorkflow',
                    'Products.GenericSetup',
                    'Products.statusmessages',
                    'zope.annotation',
                    'zope.component',
                    'zope.viewlet'],
 'meta.zcml': ['Products.CMFCore',
               'Products.GenericSetup',
               'zope.component',
               'zope.viewlet']}

The calling code in z3c.autoinclude then makes sure that first the meta.zcml is loaded for the found packages, and then the configure.zcml (or overrides.zcml in case of includeDependenciesOverrides).

Note that in my PRs I might not use the exact same order as previously. z3c.autoinclude gets the dependencies from pkg_resources, and I am not sure if the order there is guaranteed the same on all operating systems and in all setuptools versions. I have shifted the order sometimes, for example loading plone.x before plone.app.x.

[mauritsvanrees]

I mentioned that we probably need to include this somewhere:

<include package="plone.resource" file="meta.zcml" />

Otherwise a package that uses plone:static will cause an error on startup:

zope.configuration.exceptions.ConfigurationError: ('Unknown directive', 'http://namespaces.plone.org/plone', 'static')
    File "/Users/maurits/tmp/ploneenv/parts/instance/etc/site.zcml", line 17.2-17.23
    File "/Users/maurits/tmp/ploneenv/lib/python3.7/site-packages/Products/CMFPlone/configure.zcml", line 19.2-19.44
    File "/Users/maurits/tmp/ploneenv/lib/python3.7/site-packages/plone/app/discussion/configure.zcml", line 32.4-32.34
    File "/Users/maurits/tmp/ploneenv/lib/python3.7/site-packages/plone/app/discussion/browser/configure.zcml", line 133.4

This depends on the order in which packages are loaded. In current Plone 5.2, only plone.staticresources explicitly includes plone.resource:

<include package="plone.resource" />

This only includes the configure.zcml file, instead of the needed meta.zcml file, but plone.resource takes care of this, so it's okay.

But if autoinclude is not used, and plone.staticresources is loaded later than another package that use plone:static, then you get that error.
The following core packages use plone:static, so need to load plone.resource (please check the box when the PR is merged):

• [x] [plonetheme.barceloneta](https://github.com/plone/plonetheme.barceloneta/pull/185)
• [x] [plone.app.discussion](https://github.com/plone/plone.app.discussion/pull/161)
• [x] [plone.app.event](https://github.com/plone/plone.app.event/pull/311)
• [x] [plone.app.theming](https://github.com/plone/plone.app.theming/pull/164)
• [x] plone.staticresources (already done)

[mauritsvanrees]

I have made PRs for the plone.resource changes and am testing them in combination:

• [x] [5.1](https://jenkins.plone.org/job/pull-request-5.1/131/) (only plonetheme.barceloneta and plone.app.event)
• [x] [5.2 Python 2.7](https://jenkins.plone.org/job/pull-request-5.2/548/)
• [x] [5.2 Python 3.6](https://jenkins.plone.org/job/pull-request-5.2-3.6/516/)
• [x] [5.2 Python 3.7](https://jenkins.plone.org/job/pull-request-5.2-3.7/520/)

[jensens]

@mauritsvanrees thanks for this! I plan to work on pip install Plone at the conference sprint. If all works then, I'll start documenting it.

[mauritsvanrees]

Notes when I try pip install Products.Plone now, with all the above fixes:

$ virtualenv -p python3.7 ploneenv
$ cd ploneenv
$ . bin/activate
# Assume there is a buildout-coredev checkout in ~/community/plone-coredev/py3
# which has the generated constraints.txt.
$ pip install -r ~/community/plone-coredev/py3/requirements.txt  # for zc.buildout and setuptools
$ pip install -c ~/community/plone-coredev/py3/constraints.txt Products.CMFPlone
$ cat dev-requirements.txt
# Fixes for includeDependencies:
-e [email protected]:plone/plone.app.contenttypes.git@master#egg=plone.app.contenttypes
-e [email protected]:plone/plone.app.iterate.git@master#egg=plone.app.iterate
-e [email protected]:plone/plone.app.linkintegrity.git@master#egg=plone.app.linkintegrity
-e [email protected]:plone/plone.restapi.git@master#egg=plone.restapi
# Fixes to include plone.resource zcml:
-e [email protected]:plone/plonetheme.barceloneta.git@master#egg=plonetheme.barceloneta
-e [email protected]:plone/plone.app.discussion.git@master#egg=plone.app.discussion
-e [email protected]:plone/plone.app.event.git@master#egg=plone.app.event
-e [email protected]:plone/plone.app.theming.git@master#egg=plone.app.theming
$ pip install -r dev-requirements.txt
$ cat buildout.cfg 
[buildout]
# extends = https://dist.plone.org/release/5.2.0/versions.cfg
extends = https://raw.githubusercontent.com/plone/buildout.coredev/5.2/versions.cfg
parts = instance
# Use local dirs, so we can check which eggs are still installed by buildout:
eggs-directory = eggs
download-cache = downloads
abi-tag-eggs = false

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Products.CMFPlone
user = admin:admin

[versions]
# See dev-requirements.txt.  Could use mr.developer.
plone.app.contenttypes =
plone.app.iterate =
plone.app.linkintegrity =
plone.restapi =
plonetheme.barceloneta =
plone.app.discussion =
plone.app.event =
plone.app.theming =
$ bin/buildout
$ bin/instance fg
Traceback (most recent call last):
  File "bin/instance", line 11, in <module>
    import plone.recipe.zope2instance.ctl
ModuleNotFoundError: No module named 'plone.recipe'
$ ls -1 eggs
Paste-3.1.0-py3.7.egg
plone.recipe.zope2instance-6.4.0-py3.7.egg
zc.recipe.egg-2.0.7-py3.7.egg
$ rm -rf eggs/ downloads/
$ pip install -c ~/community/plone-coredev/py3/constraints.txt Paste plone.recipe.zope2instance zc.recipe.egg
$ bin/buildout
$ ls -1 eggs
$ ls -1 downloads/dist
$ bin/instance fg
...
Serving on http://0.0.0.0:8080

Success!

[mauritsvanrees]

Note: this does not say anything about how well this works with add-ons. But the idea for add-ons is:

1. Do not use includeDependencies or includeDependenciesOverrides in your configure.zcml. Instead, explicitly include the zcml of those packages.
2. It is okay to register a z3c.autoinclude.plugin entry point in your setup.py. But this may not work. In that case: in the buildout config you should add your add-on package to the zcml option in the instance part.
3. It may help to be a good citizen by including the needed (meta) zcml from packages that you use, for example <include package="plone.resource" /> when you use plone:static, and <include package="plone.behavior" file"meta.zcml /> when you use plone:behavior. If you forget this, and you see no problems, then this is only because you are lucky that some other package has done this for you.

[gotcha]

🎉

Thanks @mauritsvanrees

Is this the first step to http://bubblenet.be/blog/lets-get-rid-of-z3c.autoinclude ? 😉

[gotcha]

3. It may help to be a good citizen by including the needed (meta) zcml from packages that you use, for example `<include package="plone.resource" />` when you use `plone:static`, and `<include package="plone.behavior" file"meta.zcml />` when you use `plone:behavior`. If you forget this, and you see no problems, then this is only because you are lucky that some other package has done this for you.

If you use test layers, you should find out soonish that those meta.zcml includes are missing.

[mauritsvanrees]

Is this the first step to http://bubblenet.be/blog/lets-get-rid-of-z3c.autoinclude ? 😉

Hah, that post is almost seven years old (adding /modified to the url tells me this). We are getting there. :-D Several other steps have been made long before, but this seems to do the trick for core Plone.

I would still say it is fine for an add-on (and for core Plone packages) to use the autoinclude plugin in setup.py, and have Plone automatically load the ZCML for those packages, like in my step 2 above. The package explicitly says "please include my zcml", and this avoids double registration in the instance eggs and zcml options.
Would you agree with that?

If not, with the above fixes you should be able to set the disable-autoinclude in your site zcml and not have startup errors.
BTW, I wonder if that would make startup faster. If you have lots of non-Plone packages in your Python (maybe because you do not use a virtualenv), it may be noticeable.

[gotcha]

I would still say it is fine for an add-on (and for core Plone packages) to use the autoinclude plugin in setup.py, and have Plone automatically load the ZCML for those packages, like in my step 2 above. The package explicitly says "please include my zcml", and this avoids double registration in the instance eggs and zcml options.
Would you agree with that?

I could agree for add-ons, even if the hassle of adding two lines instead of one seems pretty light. But not for Plone core packages. Those should be explicitely included by Plone.

If not, with the above fixes you should be able to set the disable-autoinclude in your site zcml and not have startup errors.
BTW, I wonder if that would make startup faster. If you have lots of non-Plone packages in your Python (maybe because you do not use a virtualenv), it may be noticeable.

Interesting, might be another good reason to avoid autoinclude.

[jensens]

BTW, I wonder if that would make startup faster. If you have lots of non-Plone packages in your Python (maybe because you do not use a virtualenv), it may be noticeable.

While optimizing YAFOWIL we found that the whole entry-point infrastructure and esp. iter_entry_points is very slow and it is a good idea to not iterate often over them. Since here its just one iteration (I hope so at least) it should not impact the whole that much.

[mauritsvanrees]

I could agree for add-ons, even if the hassle of adding two lines instead of one seems pretty light. But not for Plone core packages. Those should be explicitely included by Plone.

It indeed seems superfluous to register e.g. plone.app.discussion with autoinclude, when CMFPlone has it in its dependencies and explicitly loads its zcml.

It gets tricky when a package starts life outside of core (like plone.app.discussion and plone.app.contenttypes did), then gets included in core, and later is pushed out of core again. And 'in core' might mean in the setup.py of Products.CMFPlone, or only in Plone.

But yes: when a package either enters or leaves core, some changes will always be needed, and this would be one of them.

I am not in a hurry to clean this up, but it would be fine to do.