Products.cmfplone: Improve Zope for a better Plone

[pbauer]

This is a collection of ideas how to improve, simplify and cleanup Zope.

The list was assembled in discussions with several people since the Plone-Community decided to take the maintenance and further development of Zope into their own hands and is neither a roadmap nor a PLIP. Once something is being worked on (some of these must probably go through the PLIP-process) it needs its own ticket. Some items might already have their own ticket or are already resolved. Feel free to point those out.

Please discuss and add further ideas here.

Merge features of CMF into Plone:

• types bases
• versioning
• product installation
• PlonePAS monkey patches
• UUID implementation

• GS handlers to work directly at the Plone level

  Publisher/Request Handling

• ADD: IPv6 support (Or stop trying to do anything below the HTTP level ourselves…)

• REMOVE: WebDAV support
• REMOVE: FTP support
• REMOVE: SOAP support
• REMOVE: XMLRPC support
• REMOVE: exotic HTTP verb support (conflicts with next?)
• ADD: built-in http verb support (instead of plone.restapi having to hack it in)
• ADD: support for more view predicates
• REMOVE: mysterious request actions for special request GET and POST params
• REMOVE: Automatic publishing of objects that have docstrings (or maybe never publish methods)
• REMOVE: Publishing of things unless they are registered as views or reached via getitem or a traversal adapter.
• REMOVE: magical bobo method and attribute usage (no one knows what bobo is)
• REMOVE: automatic marshaling of GET/POST param values (things like paramname:list)
• ADD: support for virtual hosting configuration via environment variables
• ADD: Better hooks for Zope error handling
• ADD: __parent__ pointer storage to OFS so it’s possible to know the context for a random content item pulled out of the database.
• REMOVE: implicit Acquisition

• real WSGI support (current implementation is okay but it needs help to work with plone I believe)

  Authorization/Authentication/Security

• REMOVE: Restricted Python and TTW

• REMOVE: Use of basic auth
• Merge CSRF protection
• zope root authentication isn’t secure

• Lots of write on read

  Other

• Five: Can we drop this name?

• Update or remove the ZMI

• Python 3 compatibility

  Long-term/Wishful Thinking?

• Replace the Request and Response objects

• Building a path toward Pyramid?
• Rework plumbing in the Zope publisher as a Pyramid traverser
• Initially Zope views would be completely separate from Pyramid views though later work could be done to try and unify the view
• The same for authentication/authorization/security. Later that would be completely dropped for Pyramid’s.
• Resurrect the WebOb-compatible Zope2 request/response object attempt from long ago. Webob request could be adapted to a Zope request in the traverser and the response to a webob response to begin with.

• Add Pypy compatibility

  Noteable Sprints

• Alpine City Sprint, 25.-29.01.2016

• Berlin City Sprint, 16.-20.05.2016

[gforcada]

IMHO _Python 3 compatibility_ is by no means Other but a main topic in itself.

Getting newer Zope packages (Zope PLIP) is the first step towards that direction.

[tomgross]

ADD: HTTP2.0 Support (might come with WSGI OOTB)
REMOVE: Restricted Python and TTW
Why? This is still one a very valuable and flexible feature for quick maintainance and/or complex catalog query tasks. Maybe a better mechanism to sync Python code to a VCS would be a good option. But please don't drop TTW.

[tomgross]

Python 3 compatibility ticket:
https://github.com/zopefoundation/Zope/issues/39

[ebrehault]

@pbauer when you say "REMOVE" you mean remove like "remove it from Zope itself, and implement it in a different place", or like "just remove it, period" :) ?
Because I consider very useful those items:

• WebDAV and FTP: they are very handy (my customers use it everytime they want to move a lot of files),
• RestrictedPython: it is needed to support Python expressions in TAL, and it is a key component for any advanced TTW tooling.

[thet]

I think this list doesn't reflect the community consens. IIRC, most of the items come from a discussion between @esteele and @vangheem .
I'd also love to see WebDAV support in Plone in the future - not least because I want CalDAV too :)
Is it currently possible to secure FTP support? And WebDAV? Doing FTP on unencrypted connections these days feels scary.

Regarding RestrictedPython - it's not easy to bring that to Python 3 and might be one of the major hurdles to get to that. @loechel began investigation on that at the Alpine City Sprint. Doing that needs some budget, time and someone with experience in compiler creation. Also we have to take extra special care on security issues with that.
Regarding @loechel the current implementation has also some problems.

+1 for HTTP2 support. Once we have that, JavaScript/CSS compile mechanisms could get obsolete.
@tomgross AFAIK there is currently no HTTP2 capable WSGI server. There were only SPDY efforts. But I guess time will solve this from alone :)

[pbauer]

@thet is right. This list is neither mine nor the consenus of the community but only ideas that have been thrown around. The point of this ticket is to kick off a discussion about these and others and come up with a final list of tasks that we think are worth doing.

[ebrehault]

WebSocket would be nice.
It implies to have an event loop on the server, so it would probably live outside of Zope itself.

[mauritsvanrees]

REMOVE: Automatic publishing of that have comments automatically (or maybe never publish methods)

I think you meant to say:

REMOVE: Automatic publishing of objects that have docstrings (or maybe never publish methods)

I have updated the text in the issue.

On that subject, I will post some paragraphs that I sent to the security list last year when we were working on a hotfix. Yes, this is safe to share. The rest of the comment is the complete quote.

---

I have heard that people want to rip out automatically publishing items with docstrings. That has led me to think that the zope publisher works like this:

1. Find an object.
2. Initially decide due to security settings that the item should NOT be published.
3. See that the item has a docstring, so publish it anyway.

So in this case a docstring subverts/overrides other security settings.
But I see this is not actually the case. For some items explicit traversal is done (views, ++namespaces) and a docstring is not required there. But for all other cases, a docstring is required: without a docstring Zope refuses to publish it.
Out of curiosity, at the point where we check if an item has a docstring, I changed it to always raise Forbidden, whether there is a docstring or not. Results are as expected. Want to see the Plone Site root? No way. Want to load a file? Forget it.
So there is no way to remove the lines that bless docstrings, as I thought a future plan was.

Out of more curiosity, I decided to see what would happen if I disallow access to most methods or functions. In ZPublisher/BaseRequest.py you then get this code:

        doc = getattr(subobject, '__doc__', None)
        if not doc:
            raise Forbidden(
                "The object at %s has an empty or missing " \
                "docstring. Objects must have a docstring to be " \
                "published." % URL
                )
        import types
        subobject_name = getattr(subobject, '__name__', '')
        if (subobject_name not in
                (
                    # resources, files, images, lots of objects:
                    'index_html',
                    # webdav/ftp:
                    'manage_DAVget',
                    'manage_FTPget',
                    'PUT',
                    'PROPFIND',
                    # robotframework:
                    'run_keyword',
                ) and
                # ZMI:
                not subobject_name.startswith('manage_') and
                # robotframework:
                not subobject_name.startswith('get_keyword')
                ):
            if isinstance(subobject, types.MethodType):
                err = "Not allowed to access method {0} for {1} ({2})".format(
                    subobject_name, URL, subobject)
                print(err)
                raise Forbidden(err)
            if isinstance(subobject, types.FunctionType):
                err = "Not allowed to access function {0} for {1} ({2})".format(
                    subobject_name, URL, subobject)
                print(err)
                raise Forbidden(err)

More methods would need to be whitelisted. But with this I don’t notice difficulties browsing the site. (Plone 4.3 coredev.) CMFPlone Robot tests pass. A few test failures in testSecurity and testCSRFProtection, where we get a 404 error instead of 302 or 403, which is logical. I was busy with plone.app.imaging, which then gives a test failure because the transformImage method of ATContentTypes is called, which then fails, meaning the transform tab on images fails. Obvious fix would be to make that a view instead.

This is not a serious suggestion for a hotfix, but maybe it gets people thinking about what it would take to get rid of the docstring madness.

Bed time reading: http://docs.zope.org/zope_secrets/index.html

[gforcada]

@mauritsvanrees thanks for sharing this excellent report about docstrings and publishing (which I always wondered about).

If I get it right though, converting everything to views and shrinking more and more the ZMI would make this code less and less important right?

I mean, if we want to get rid of the ZMI, we still need a way to create a Plone instance, but once that's solved, nothing else (if everything is a view) should need this auto-magic-publishing

[mauritsvanrees]

That sounds about right.

Well, those views will still use templates and they might use something like tal:condition="context/portal_membership/someMethod", which would need a docstring and the blessing of the zope publisher. Arguably the someMethod call would need to be done in the view code.

Of course there are still lots of things in the ZMI that do not yet have a counterpart in the Plone UI, but some are plipped already, like being able to manage portal_actions.

[gforcada]

The tal:condition sounds like good material for a plone.recipe.codeanalysis plugin, reducing these while at the same time reducing our dependency of ZMI would mean we could eventually turn off the autopublishing.

[hvelarde]

for those who don't know what bobo is (or was), read A Bit of History.

[gforcada]

@hvelarde more of a reason to get rid of it, only mentioned on the side of Zope technology overview :-)

[thet]

The browser:view zcml directive supports allowed_attributes, which configures individual methods on a view to be accessible through the web. For these methods, you also need docstrings as I learned here: https://github.com/collective/collective.lineage/commit/87387af2d43f5648ba8928df6a62e1c3669903a5

[gforcada]

@thet that's interesting, because we could make that zcml directive the only way to auto-publish attributes without the need of docstrings (although that's only for views, not for objects)...

[hvelarde]

I'm not so sure about adding support for HTTP/2; even as it seems to solve the problem of many request (and our problem with resource compilation) there is no clear consensus on the protocol's advantages and some people are very critical with it.

@tomgross I recommend you to read this Poul-Henning Kamp (Varnish) piece on it: HTTP/2.0 — The IETF is Phoning It In.

besides that, we can always use a proxy like nginx on front to support such feature without having to develop that for Zope.

[datakurre]

@ebrehault We do have event loop in Zope2 and I did try websocket server (collective.zws) on it :) Unfortunately (unsurprsingly), there was no maintained websocket server on top of asyncore and the design of asyncore would probably have issues with large amount of concurrent websocket connections.

[miohtama]

oblig. meme GIF

[miohtama]

@hvelarde You are right; front end internet facing web server like Nginx should handle HTTP/2. Not sure if application servers can benefit of it that much (unless Plone will run on some Python application server which directly supports it).

[zopyx]

REMOVE: WebDAV support
REMOVE: FTP support

Leave this! both protocols are still widely used by various installations. Removing them requires major changes to other systems relying on this integration layer with Plone.

[agitator]

I'm with @zopyx leave WebDAV/FTP where it is!
Even if it could work better...

[idgserpro]

REMOVE: Restricted Python and TTW
Why? This is still one a very valuable and flexible feature for quick maintainance and/or complex catalog query tasks. Maybe a better mechanism to sync Python code to a VCS would be a good option. But please don't drop TTW.

This is valuable as well for quick maintainance/fix in some companies where you don't have ssh access or provision/deploy for imediate fix, having to rely in custom code or portal_view_customizations for fast fixes.

[davilima6]

portal_view_customizations has been broken since day 0 and nowadays it seems to support even less usecases, e.g.: #1632. I'd be +1 for removing it. After that we could put more energy into collective.jbot, for instance.

[tomgross]

@thet There are HTTP/2 ready WSGI implementations meanwhile:

• https://netius.hive.pt/
• https://python-hyper.org/projects/h2/en/stable/wsgi-example.html