Products.cmfplone: PLIP: Using Dexterity for Plone Site

[jaroel]

PLIP (Plone Improvement Proposal)

Responsible Persons

Proposer: Roel Bruggink

Seconder: Alessandro Pisa

Abstract

Replace the CMFPlone Portal class with a Dexterity implementation.

Motivation

Moving from a CMF item to a Dexterity container will allow us to edit the site root directly using the DX framework including support for behaviors.

This makes it possible to add the rich text behavior to the site root. This way we don’t need to use extra documents to select as default views. This is the last step we need to be able to go with a full folderish objects site and get rid of the “select item as default view” which is a huge usability improvement

Assumptions

Proposal & Implementation

Replace the base class for Products.CMFPlone.Portal.PloneSite with a DX container.
Development is initially done on the latest stable (5.1) branch of Products.CMFPlone. We'll move to unstable (5.2) when this is done or if 5.2 is released, whichever is earlier.

Deliverables

PLIP buildout config can be found here: https://github.com/plone/buildout.coredev/blob/6.0/plips/plip-2454-dx-siteroot.cfg

Jenkins job: https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.8/

Change Products.CMFPlone.Portal.PloneSite from CMF to DX. #2612
Various supportive changes in -at least- https://github.com/plone/plone.app.contenttypes/pull/475 and https://github.com/plone/plone.dexterity/pull/85.

Risks

Currently unknown.

Participants

Roel Bruggink
Alessandro Pisa

[jensens]

How do you plan to migrate the Site Root?

[jaroel]

I haven't given it too much thought atm.
I copied over an existing Plone 5.1 Data.fs into my development stuff and at least it loaded the state correctly from the ZODB, had the correct mro for the Plone Site root and rendered the site just fine.

Something like this should do using a view/script on the zope root for each Plone site:

class MigrateRootToDx(BrowserView):

    def __call__(self):
        from Acquisition import aq_base
        from plone.protect.auto import safeWrite
        request = self.request

        print("Start migration Plone Sites root to DX")
        for site in self.context.objectValues('Plone Site'):
            safeWrite(site, request)
            print("Running for {}".format(site))

            # Initialise the btree.
            if '_tree' not in site.__dict__:
                print("Initialising btrees")
                site._initBTrees()

            # Migrate existing portal content.
            print("Migrate content")
            for obj_meta in site._objects:
                obj_id = obj_meta['id']
                print("Migrating object {}".format(obj_id))
                # Load the content item, tool, or whatever
                obj = getattr(site, obj_id)
                # insert it into the btree.
                # Use _setOb so we don't reindex stuff. the paths stay the same.
                site._setOb(obj_id, aq_base(obj))
                # Drop the attribute
                del site.__dict__[obj_id]

            # Drop no-longer needed bookkeeping
            print("Clean up bookkeeping data")
            del site.__dict__['_objects']
            site._p_changed = True

I do need to check what bookkeeping can be dropped.

edit: I've just tested the above and that seems to Just Works (tm).

>>> sorted(app.Plone.keys())
['_Access_arbitrary_user_session_data_Permission', '_Access_contents_information_Permission', '_Access_inactive_portal_content_Permission', '_Access_session_data_Permission', '_Add_portal_content_Permission', '_Add_portal_folders_Permission', '_Add_portal_member_Permission', '_Allow_sendto_Permission', '_CMFEditions__Access_previous_versions_Permission', '_CMFEditions__Apply_version_control_Permission', '_CMFEditions__Checkout_to_location_Permission', '_CMFEditions__Revert_to_previous_versions_Permission', '_CMFEditions__Save_new_version_Permission', '_Change_local_roles_Permission', '_Content_rules__Manage_rules_Permission', '_Copy_or_Move_Permission', '_Delete_comments_Permission', '_Delete_objects_Permission', '_Delete_own_comments_Permission', '_Edit_comments_Permission', '_FTP_access_Permission', '_List_folder_contents_Permission', '_List_portal_members_Permission', '_List_undoable_changes_Permission', '_Mail_forgotten_password_Permission', '_Manage_properties_Permission', '_Modify_portal_content_Permission', '_Modify_view_template_Permission', '_Portlets__Manage_own_portlets_Permission', '_Portlets__Manage_portlets_Permission', '_Portlets__View_dashboard_Permission', '_Reply_to_item_Permission', '_Request_review_Permission', '_Review_comments_Permission', '_Review_portal_content_Permission', '_Search_ZCatalog_Permission', '_Set_own_password_Permission', '_Set_own_properties_Permission', '_Show_Toolbar_Permission', '_Undo_changes_Permission', '_Use_Database_Methods_Permission', '_Use_external_editor_Permission', '_Use_mailhost_services_Permission', '_Use_version_control_Permission', '_View_Groups_Permission', '_View_History_Permission', '_View_Permission', '_View_management_screens_Permission', '_WebDAV_Lock_items_Permission', '_WebDAV_Unlock_items_Permission', '_WebDAV_access_Permission', '__ZCacheManager_ids__', '__ac_local_roles__', '__ac_roles__', '__allow_groups__', '__annotations__', '__before_publishing_traverse__', '__before_traverse__', '__error_log__', '_components', '_count', '_mt_index', '_owner', '_plone_app_contenttypes__Add_Collection_Permission', '_plone_app_contenttypes__Add_Document_Permission', '_plone_app_contenttypes__Add_Event_Permission', '_plone_app_contenttypes__Add_File_Permission', '_plone_app_contenttypes__Add_Folder_Permission', '_plone_app_contenttypes__Add_Image_Permission', '_plone_app_contenttypes__Add_Link_Permission', '_plone_app_contenttypes__Add_News_Item_Permission', '_plone_app_event__Import_Ical_Permission', '_plone_app_multilingual__Manage_Translations_Permission', '_plone_resource__Export_ZIP_file_Permission', '_plone_resourceeditor__Manage_Sources_Permission', '_properties', '_tree', '_v__providedBy__', 'cmf_uid', 'contributors', 'creation_date', 'creators', 'default_page', 'description', 'effective_date', 'expiration_date', 'format', 'id', 'language', 'modification_date', 'rights', 'subject', 'title']

>>> app.Plone.portal_catalog
<CatalogTool at /Plone/portal_catalog>

>>> app.Plone.portal_catalog()
[<Products.ZCatalog.Catalog.mybrains object at 0x10b27e598>, <Products.ZCatalog.Catalog.mybrains object at 0x10b27e0b8>, <Products.ZCatalog.Catalog.mybrains object at 0x10b258668>, <Products.ZCatalog.Catalog.mybrains object at 0x10b258940>, <Products.ZCatalog.Catalog.mybrains object at 0x10b258bb0>]

>>> app.Plone.portal_catalog()[0]
<Products.ZCatalog.Catalog.mybrains object at 0x10b258328>

>>> app.Plone.portal_catalog()[0].getObject()
<Document at /Plone/front-page>

>>> 'portal_catalog' in app.Plone.__dict__
False

[jaroel]

We'll need to add the Plone Site FTI and fix the edit method alias when upgrading.
Need to check plone.app.multilingual as we won't have a front-page object anymore.

[ebrehault]

@jaroel The FWT is very positive about this PLIP.

It would be useful to get more information about the motivations (what could be the benefits? What kind of behaviors could be usefully used on the Plone Site object) as it would help to communicate about this important improvement.

Note: we will vote on the PLIP during our next meeting.

[sunew]

for one, selectablecontrainstypes behavior would be super useful:)

[robgietema]

This makes it possible to add the rich text behavior to the site root. This way we don’t need to use extra documents to select as default views. This is the last step we need to be able to go with a full folderish objects site and get rid of the “select item as default view” which is a huge usability improvement. I can second this proposal.

[ebrehault]

@jaroel, the @plone/framework-team approved this PLIP

[jaroel]

@ebrehault that's great news, thanks!

[tkimnguyen]

How does this work? "This is the last step we need to be able to go with a full folderish objects site and get rid of the 'select item as default view'" In CastleCMS we got rid of default items by using Mosaic and including a rich text tile on the Folder content type default layout

[tisto]

@jaroel would you mind giving us an update on the current status? What is missing before we can review your code? Do you plan to integrate this in Plone 5.2 or later?

[tisto]

@tkimnguyen in Plone-React we got rid of default items as well. The new tiles-based "Page" type is folderish by default as well.

[pbauer]

@tkimnguyen see https://github.com/plone/plone-react/blob/master/api/buildout.cfg#L24

[jaroel]

@tisto In practice it seems to work just fine. I'm resolving the test failures when I have time.
See https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot/ for the current state.

You can check the changes done so far using the PLIP buildout config https://github.com/plone/buildout.coredev/blob/5.1/plips/plip-2454-dx-siteroot.cfg .

I wouldn't see this as a blocker for 5.2. I'd not even consider this for 5.2, unless people have used this in production somewhere.

[jaroel]

@gforcada or @tisto I'm getting timeout accessing the ftp server on jenkins (https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot/12/testReport/plone.app/testing/layers_rst/)
Any idea what could be going on there?

[jaroel]

FYI I'm back on this. Primary goal is to make this work for 5.2+, as that seems stable enough now.

[tisto]

@jaroel we do not plan to do a 5.3 release so this would be Plone 6 then. I added the label...

[jaroel]

@jensens FYI:

1. So far bin/instance starts and I have a functional Plone site root.
2. The tests failures from https://github.com/plone/plone.supermodel/commit/c1822bb822270b323558bfd3cd49229c50b04140 are back
3. https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.8/ has some failures
4. The ZMI (/manage) doesn't reflect the normal interface for the site root
   5. The Products.CMFEditions failures make me sad. They have disappeared somehow, which makes me sad

[esteele]

What's the current status of this? It looks like tests are currently green.

[jaroel]

I've just updated the branches in the repos I needed.
The "upgrade script" https://github.com/plone/Products.CMFPlone/issues/2454#issuecomment-400022190 should be added somewhere. This could be in Products.CMFPlone.Portal.__getstate__ or as a normal upgrade step.
The FWT should review what has been done, and people must test it with actual production data.

To be honest I don't really have time nor incentive currently to work on this except for once in a blue moon. I'm fairly certain it would be more effective if someone like @ale-rt would champion this PLIP.

[ale-rt]

I will work on this at the Not-an-Alpine-City-Sprint

[jaroel]

@ale-rt I'm sure this is in very capable hands, thank you very much Alessandro!
I'm more than happy to answer questions.

[esteele]

@ale-rt I see you've taken care of the upgrade step. Is this ready for FWT review? If not, are there additional tasks I (or others) can help out with?

[ale-rt]

Hi @esteele! I am planning to recheck this on the 23rd of April (there is a Zope sprint scheduled).
I am not using this on production yet but so far I did not spot any issue.
Hi really would like somebody from the @plone/framework-team to pick this up and review it.

[ale-rt]

And of course testing by anybody: if anyone can check that their DB run fine after this switch it would be super cool.

[pbauer]

I tested it against a Plone 6 (master) database with p.a.multilingual and that failed initially with expected tracebacks and I could not open the portal in the ZMI.

After accessing http://localhost:8080/Plone/portal_setup/manage_upgrades I managed to run the two upgrade-steps "Be sure that the Plone Site FTI is a dexterity one" and "Make the Plone Site a dexterity container". Then the site seemed to work fine only the FTI of Plone Site has lost the default_view property language-switcher and was reset to listing_view.

What freaks me out is that some test failed in unexpected ways: The content of [i for i in portal._tree] was not the same as portal.keys() and portal.contentValues() failed with:

>>> m.contentValues()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/pbauer/workspace/dynajet_relaunch/src-mrd/Products.CMFCore/src/Products/CMFCore/PortalFolder.py", line 202, in contentValues
    return [item[1] for item in self.contentItems(filter)]
  File "/Users/pbauer/workspace/dynajet_relaunch/src-mrd/Products.CMFCore/src/Products/CMFCore/PortalFolder.py", line 188, in contentItems
    return self._filteredItems(ids, filter)
  File "/Users/pbauer/workspace/dynajet_relaunch/src-mrd/Products.CMFCore/src/Products/CMFCore/PortalFolder.py", line 174, in _filteredItems
    obj = get(id)
  File "/Users/pbauer/.cache/buildout/eggs/plone.folder-3.0.3-py3.8.egg/plone/folder/ordered.py", line 73, in _getOb
    raise AttributeError(e)
AttributeError: 'de'

[ale-rt]

Thanks @pbauer for reporting this back.
About the upgrade, were you able to run the http://localhost:8120/Plone/@@plone-upgrade or that was not helpful because you were already using Plone 6?
I will take care of copying the default_view and maybe some other properties during the upgrade.

[pbauer]

I already used Pone 6 master for that test but I'll test another project which still runs on Plone 5.2

[pbauer]

I just tried the upgrade using @@plone-upgrade with a project that ran 5.2.4 before and it worked like a charm. I updated the plip-branches locally to the current master of Plone 6. The project has no plone.app.multilingual.

[tisto]

@ale-rt we did not discuss this PLIP at our last FWT meeting. @mauritsvanrees would like to start doing a first (internal) release of Plone 6. Can you give us a heads up about the status here,? If needed, we could schedule a @plone/framework-team meeting on short notice to push this further in order to get this into the first release.

[mauritsvanrees]

I upgraded a Plone 6 site to dx site root and it seemed fine.
After the upgrade, the front-page content item is still there and is used as default page. I wondered if its text would be taken over, and maybe even the front-page itself removed, but this is not the case. Current state is probably best.
Creating a new site worked fine as well.

Two minor things:

1. Currently the homepage shows the text field plus the contents of the root (news, events, Members). If you only want to show the text field and no contents, there is currently no way.
2. The plone.excludefromnavigation behavior is enabled, but when you check this field in the edit form, it has no effect. We should probably just remove this behavior.

For the first one, we could make a copy of document_view and make this available in the view methods. Or is there some basic view which I am not aware of that renders all defined fields?
Probably a simple siteroot_view.pt would be fine. That also gives a nice way for integrators to override this with z3c.jbot.
For me, it is fine to do this after things get merged.

The excludefromnavigation behavior is probably best removed now, unless someone sees a need for this anyway.

[mauritsvanrees]

We could add the versioning behavior, though I don't see this having an effect when I try it, so this might need changes elsewhere.

We could add the constraintypes behavior. This seems to work well when I try it.

Both could be done later, or we leave this as an option for the user.

[mauritsvanrees]

I created two PRs that were missing. Related PRs are now:

• [ ] https://github.com/plone/Products.CMFPlone/pull/2612
• [ ] https://github.com/plone/plone.app.contenttypes/pull/475
• [ ] https://github.com/plone/plone.app.upgrade/pull/256

Plus the following PRs which can maybe be merged separately because they seem fine for current Plone as well:

• [x] https://github.com/plone/plone.restapi/pull/1202
• [x] https://github.com/plone/plone.app.multilingual/pull/336
• [x] https://github.com/plone/plone.app.testing/pull/60
• [x] https://github.com/plone/plone.dexterity/pull/85
• [ ] https://github.com/plone/plone.supermodel/pull/27

If we merge the PRs which can be done separately, it makes the rest of the PLIP smaller.
[Update: most of these have been merged.]

Note that some branches were created in 2019, and master has been merged into them several times, which makes for an ugly and confusing history. So it could be an option to create fresh branches, or do a rebase. I did a fresh branch for plone.restapi, as the only change left was in two lines.

[mauritsvanrees]

I have updated the PLIP config to only test the three remaining packages with branches.
I have started the three PLIP jobs on Jenkins, on 3.7, 3.8 and 3.9.
I have deleted the 3.6 PLIP project, because Plone 6 no longer supports it.

[mauritsvanrees]

The three jobs all have five the same failures in the robot tests. So this may need some work. Locally the failing ones pass though.
The 3.7 job even has 34 robot failures, I have restarted this one.

See the 3.9 job.
Four are in the user schema editor, where for a checkbox we get:

is not clickable at point (1771, 2208).
Other element would receive the click: <div class="plone-toolbar-container">

See the screenshot, which looks very wrong: the toolbar is visible inside a popup/overlay. When I try it locally, it looks fine though.

The fifth is for the scenario "A page is opened to edit in TinyMCE" where we get:

Button with locator 'css=.pattern-relateditems-container button.favorites' not found.

See screenshot

[mauritsvanrees]

I have restarted the 3.7 job and it again had the same 34 robot failures.
I have restarted the 3.8 job and it again had the same 5 robot failures.
Jenkins did not do much at the time.
So there are no random failures, but real ones.

[mauritsvanrees]

Sorry, I lost sight of the plone.supermodel PR. I added it to the list above now. Can be merged separately. @ale-rt has re-added it to the plip config file and this should fix the PLIP build.

[ale-rt]

Yes, all the branches are green right now:

• https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.7/
• https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.8/
• https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.9/

[ale-rt]

@ale-rt we did not discuss this PLIP at our last FWT meeting. @mauritsvanrees would like to start doing a first (internal) release of Plone 6. Can you give us a heads up about the status here?

@tisto this PLIP gets periodically green.

For me it is even good to be merged now, but nobody did a formal review of the 4 branches:

• https://github.com/plone/Products.CMFPlone/pull/2612
• https://github.com/plone/plone.app.upgrade/pull/256
• https://github.com/plone/plone.supermodel/pull/27
• https://github.com/plone/plone.app.contenttypes/pull/475/

I never used this code on production, but using this plip branch I could easily migrate and run simple Plone 5.2 websites.
If you ask me I would merge all the PRs right now (this will target the first alpha version...).
I would also like to hear other people opinions.

If needed, we could schedule a @plone/framework-team meeting on short notice to push this further in order to get this into the first release.

Unluckily I think some FWT people are now on vacation, that is why we agreed to meet next time the 7th of September (see https://community.plone.org/t/fwt-meeting-minutes-2021-07-20/14110).
Anyway if other @plone/framework-team people want to organize a meeting on a short notice I will be fine with that :)

[tisto]

@ale-rt let's try to get a formal review of the four branches and feedback from as many @plone/framework-team members as possible. I'd be open to schedule a call on short notice (tomorrow?).

Maybe if all available FWT members and the release managers @mauritsvanrees @esteele agree, we can merge this PLIP and make a holiday exception.

I guess the only way to properly test this is to get it into Plone 6 alpha.