Products.cmfplone: Plone should not create temp_folder

[mauritsvanrees]

In a standard Plone buildout, you will get the following snippet in parts/instance/etc/zope.conf that uses Products.TemporaryFolder to create a temporary storage for sessions:

<zodb_db temporary>
    # Temporary storage database (for sessions)
    <temporarystorage>
      name temporary storage for sessioning
    </temporarystorage>
    mount-point /temp_folder
    container-class Products.TemporaryFolder.TemporaryContainer
</zodb_db>

Core Plone has not used this temporary storage since Plone 4.0. But add-ons might.
I ran into problems myself when I tried using this in Plone 5.1. See explanation by Dieter Maurer in that same issue, pointing out why this tempstorage is not a good idea.

Zope is getting rid of the Products.TemporaryStorage dependency in Products.Sessions. So should Plone.
If someone does want to store sessions in Zope, here is a good explanation of an alternative to the current temporary storage.

Currently (Plone 5.2.0):

• CMPlone has Products.TemporaryStorage in its dependencies
• plone.recipe.zope2instance has a snippet for parts/instance/etc/zope.conf that gets added by default, but can be switched off by setting zodb-temporary-storage = off.

When I keep the dependency and remove the snippet, I get these errors in the log, although Plone seems to load fine:

ERROR   [Zope.ZODBMountPoint:221][waitress] Failed to mount database.
 <class 'ZConfig.ConfigurationError'> (No database configured for mount point at /temp_folder)
Traceback (most recent call last):
  File "/Users/maurits/shared-eggs/cp27m/Products.TemporaryFolder-5.3-py2.7.egg/Products/ZODBMountPoint/MountedObject.py", line 252, in _getOrOpenObject
    conn = self._getMountedConnection(anyjar)
  File "/Users/maurits/shared-eggs/cp27m/Products.TemporaryFolder-5.3-py2.7.egg/Products/ZODBMountPoint/MountedObject.py", line 147, in _getMountedConnection
    self._getDB()
  File "/Users/maurits/shared-eggs/cp27m/Products.TemporaryFolder-5.3-py2.7.egg/Products/ZODBMountPoint/MountedObject.py", line 157, in _getDB
    return getConfiguration().getDatabase(self._path)
  File "/Users/maurits/shared-eggs/cp27m/Zope-4.1.2-py2.7.egg/Zope2/Startup/datatypes.py", line 262, in getDatabase
    name = self.getName(mount_path)
  File "/Users/maurits/shared-eggs/cp27m/Zope-4.1.2-py2.7.egg/Zope2/Startup/datatypes.py", line 279, in getName
    self._mountPathError(mount_path)
  File "/Users/maurits/shared-eggs/cp27m/Zope-4.1.2-py2.7.egg/Zope2/Startup/datatypes.py", line 256, in _mountPathError
    % mount_path)
ConfigurationError: No database configured for mount point at /temp_folder

I can remove the temp_storage object in the ZMI without problem.

The other way around, when I remove the dependency, but keep the snippet, Plone does not start up:

$ bin/instance fg
Error: unknown type name: u'temporarystorage'
(line 28 in file:///Users/maurits/community/plone-coredev/5.2/parts/instance/etc/zope.conf)

When I remove both the dependency and the snippet, I see no errors on startup and in usage, except that I cannot visit the temp_storage object in the ZMI. I can remove it there without problem.

So we should:

• [ ] Remove the Products.TemporaryStorage dependency from `Products.CMFPlone.
• [ ] In plone.recipe.zope2instance either set the zodb-temporary-storage option off by default, or remove the option totally.

Question is when to do that. This seems safe for at least 99% of users. But there could be users that actually use this feature. Chances are that they are running into trouble already, but maybe for their use case it is working fine.
The safest thing to do, is probably to wait with this until Plone 6. We can put it in the upgrade documentation then.

[mauritsvanrees]

I had a quick look at this on Plone 5.2.

• Remove the temporary storage snippet from parts/instance/etc/zope.conf (manually or via zodb-temporory-storage = off)
• Start Plone, you get an error ConfigurationError: No database configured for mount point at /temp_folder but startup continues.
• Manually remove these items from Zope root: temp_folder, session_data_manager (which uses /temp_folder in its properties), and browser_id_manager. Maybe the last one can stay, but it is created by Products.Sessions on startup, which also creates the other two objects, so I hope this one can go away too.
• Restart Plone. You see an error [Products.TemporaryFolder:55][MainThread] Could not initialize a Temporary Folder because a database was not configured to be mounted at the /temp_folder mount point. This is raised by Products.ZODBMountPoint, which is included in the distribution of Products.TemporaryStorage. But startup continues.
• The browser_id_manager and session_data_manager have been automatically recreated by Products.Sessions.

If you now do anything with request.SESSION, like in a Python Script, then you get an error:

SessionDataManagerErr: External session data container '/temp_folder/session_data' not found.

This is good: if any add-on uses this session mechanism, then you immediately see it failing, so you know something is going on (except when PAS swallows these errors).

So to really get rid of tempstorage, we must get rid of Products.TemporaryFolder and Products.Sessions.

• Products.CMFPlone/setup.py has both in its requirements.
• Both have the other in their test requirements.
• Products.PluggableAuthService/setup.py has Products.Sessions in the requirements. Apparently this is an indirect dependency for CSRF protection.

Likely the SessionAuthHelper.py from PluggableAuthService does not work, because it uses SESSION. I can add this plugin, but likely any errors are swallowed. Well, depending on which plugin interfaces you activate, I can still login as admin, but when I logout, I get the same error as above. But Plone does not use this plugin by default.
Its CSRF code is here. But I cannot get a print line in there to show anything, nor a pdb. So maybe it never gets called. I even tried in the root Zope acl_users. But maybe everything is overridden by PlonePAS.

Ah, I see: the CSRF code is monkey patched by plone.protect.

So it would help if we can make Products.Sessions an optional dependency of Products.PluggableAuthService. Or get rid if the dependency altogether.

Some other places where I see SESSION in 5.2:

• Products/CMFPlacefulWorkflow/tests/testCMFPlacefulWorkflow.py so in tests. SESSION gets replaced with a dummy session. Probably no longer needed since Plone 4 even.
• Products/contentmigration/tests/cmtc.py so in tests. Same as CMFPlacefulWorkflow.
• Products/CMFFormController/FormController.py but this is only when looping over request.keys(), and the code continues when key is REQUEST. Fine to keep this. (Except that we want to get rid of CMFFormController I think.)
• ZServer/Testing/utils.py, but this code is only executed when Products.Sessions can be imported, so that is fine.

So the biggest work is in Products.PluggableAuthService.

[mauritsvanrees]

Products.Sessions 4.8 is much friendlier when there is no Products.TemporaryFolder or no other object manager at /temp_folder/session_data. I have updated the pin in coredev 5.2.
And PluggableAuthService is getting friendlier when Products.Sessions is not setup.
So we have progress.

[mauritsvanrees]

Note: PlonePAS has code that accesses the session data manager. This is on logout, and only tries to remove a session if it is there. In Products.Sessions 4.7 or older, without a proper /temp_folder/session_data, this gives a SessionDataManagerErr: External session data container '/temp_folder/session_data' not found.. With 4.8, you only get a warning.

[icemac]

See also https://github.com/zopefoundation/Products.TemporaryFolder/issues/12 – Products.TemporaryFolder no longer creates /temp_folder either.

[mauritsvanrees]

Thanks for the pointer, @icemac.
So I guess on Plone 6 we should pin Products.TemporaryFolder and Products.ZODBMountpoint to the latest versions, if Zope 5 does not do that. Then no new items are created automatically.
And then in an upgrade step we can remove the objects from the zope root, though we should probably keep them if the packages are still there, because they may be included intentionally.

[mauritsvanrees]

Some initial code is in plone.app.upgrade branch maurits/issue-2957-delete-temp-folder-60, especially commit https://github.com/plone/plone.app.upgrade/commit/d549e7bc37663f4ab7f363e2f049c1e45aa0c511.

Definitely not ready, and needs more testing. But I had to try something out.

[tisto]

@mauritsvanrees I just ran into the "temp_folder" problem in one of our client projects (Plone 5.2.2, Plone 5.2.4 fails with the same error). When I start the instance with "bin/instance fg" I get the following trace:

2021-05-27 09:05:48,289 ERROR [Zope.ZODBMountPoint:221][MainThread] Failed to mount database. <class 'ZConfig.ConfigurationError'> (No database configured for mount point at /temp_folder) Traceback (most recent call last): File "/Users/timo/.buildout/eggs/Products.TemporaryFolder-5.3-py3.7.egg/Products/ZODBMountPoint/MountedObject.py", line 252, in _getOrOpenObject conn = self._getMountedConnection(anyjar) File "/Users/timo/.buildout/eggs/Products.TemporaryFolder-5.3-py3.7.egg/Products/ZODBMountPoint/MountedObject.py", line 147, in _getMountedConnection self._getDB() File "/Users/timo/.buildout/eggs/Products.TemporaryFolder-5.3-py3.7.egg/Products/ZODBMountPoint/MountedObject.py", line 157, in _getDB return getConfiguration().getDatabase(self._path) File "/Users/timo/.buildout/eggs/Zope-4.5.3-py3.7.egg/Zope2/Startup/datatypes.py", line 262, in getDatabase name = self.getName(mount_path) File "/Users/timo/.buildout/eggs/Zope-4.5.3-py3.7.egg/Zope2/Startup/datatypes.py", line 279, in getName self._mountPathError(mount_path) File "/Users/timo/.buildout/eggs/Zope-4.5.3-py3.7.egg/Zope2/Startup/datatypes.py", line 256, in _mountPathError % mount_path) ZConfig.ConfigurationError: No database configured for mount point at /temp_folder 2021-05-27 09:05:48,314 INFO [Zope:45][MainThread] Ready to handle requests Starting server in PID 4788. Serving on http://0.0.0.0:8080

Here is our parts/instance/etc/zope.conf:

%define INSTANCEHOME /Users/timo/workspace/xxx/api/parts/instance instancehome $INSTANCEHOME %define CLIENTHOME /Users/timo/workspace/xxx/api/var/instance clienthome $CLIENTHOME debug-mode off security-policy-implementation C verbose-security off default-zpublisher-encoding utf-8 <environment> zope_i18n_compile_mo_files true CHAMELEON_CACHE /Users/timo/workspace/xxx/api/var/cache </environment> <zodb_db main> # Main database cache-size 30000 # Blob-enabled FileStorage database <blobstorage> blob-dir /Users/timo/workspace/xxx/api/var/blobstorage # FileStorage database <filestorage> path /Users/timo/workspace/xxx/api/var/filestorage/Data.fs </filestorage> </blobstorage> mount-point / </zodb_db> python-check-interval 1000%

Our buildout configuration (with "zodb-temporary-storage = off"):

`````

============================================================================

CORE BUILDOUT TO SHARE CORE CONFIGURATION ACROSS ALL BUILDOUT CONFS

============================================================================

[buildout]
extends = https://dist.plone.org/release/5.2.3/versions.cfg
allow-picked-versions = false
show-picked-versions = true
extensions = mr.developer
auto-checkout = *
always-checkout = force

Setting a download cache prevents solr to download each time :)

download-cache = downloads
parts =
instance
code-analysis
test
scripts
omelette
robot
robot-server

[sources]
customer.policy = fs customer.policy
kitconcept.volto = git [email protected]:kitconcept/kitconcept.volto.git

kitconcept.glossary = git [email protected]:kitconcept/kitconcept.glossary.git branch=local_api

kitconcept.contentcreator = git [email protected]:kitconcept/kitconcept.contentcreator.git

plone.restapi = git git://github.com/plone/plone.restapi.git [email protected]:plone/plone.restapi.git branch=fix-resolveuid

collective.solr = git [email protected]:collective/collective.solr.git branch=master

Zope = git https://github.com/zopefoundation/Zope.git [email protected]:zopefoundation/Zope.git branch=4.x

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
http-address = 8080
eggs =
Plone
Pillow
plone.reload
customer.policy [test]
Products.PloneHotfix20210518

zcml = customer.policy

zcml-additional =
xmlns:plone="http://namespaces.plone.org/plone"
xmlns:solr="http://namespaces.plone.org/solr"
xmlns:zcml="http://namespaces.zope.org/zcml"
zcml:condition="installed plone.restapi">
allow_origin="http://localhost:3000,http://127.0.0.1:3000"
allow_methods="DELETE,GET,OPTIONS,PATCH,POST,PUT"
allow_credentials="true"
expose_headers="Content-Length,X-My-Header"
allow_headers="Accept,Authorization,Content-Type,X-Custom-Header"
max_age="3600"
/>

environment-vars = zope_i18n_compile_mo_files true
zodb-temporary-storage = off

#

START RELSTORAGE

info: create a database

psql

CREATE USER plone WITH PASSWORD 'secret';

CREATE DATABASE plone OWNER plone;

- or -

$ createuser plone

$ createdb --owner=plone plone

[relstorage]
name = $(PLONE_DB_NAME)
host = $(PLONE_DB_HOST)
user = $(PLONE_DB_USER)
keep-history = $(PLONE_DB_KEEP_HISTORY)
blob-cache = ${buildout:directory}/var/blob-cache
export-directory = ${buildout:directory}/var
eggs =
RelStorage[postgresql]
parts =
relstorage-export
relstorage-pack

See: https://pypi.python.org/pypi/RelStorage for all options

password is expected in a .pgpass file (users home)

dsn = dbname='${:name}' host='${:host}' user='${:user}'
configuration =
type postgresql
dsn ${:dsn}
keep-history ${:keep-history}
cache-local-mb $(RELSTORAGE_CACHE_LOCAL_MB)
cache-local-compression zlib
shared-blob-dir false
blob-dir {:blob-cache}
blob-cache-size 2gb

[relstorage-export]
recipe = collective.recipe.template
input = ${buildout:directory}/etc/relstorage/relstorage-export.cfg.in
output = ${buildout:directory}/relstorage-export.cfg

dsn = ${relstorage:dsn}
keep-history= ${relstorage:keep-history}
filestorage-export = ${relstorage:export-directory}/${relstorage:name}.fs

[relstorage-pack]
recipe = collective.recipe.template
input = ${buildout:directory}/etc/relstorage/relstorage-pack.cfg.in
output = ${buildout:directory}/relstorage-pack.cfg

dsn = ${relstorage:dsn}
keep-history= ${relstorage:keep-history}

END RELSTORAGE

#

[code-analysis]
recipe = plone.recipe.codeanalysis
directory = ${buildout:directory}/src
flake8-exclude = bootstrap.py,bootstrap-buildout.py,docs,*.egg.,omelette,src/collective.solr,src/kitconcept.contentcreator,src/plone.restapi
flake8-max-complexity = 40
flake8-max-line-length = 1000
flake8-ignore = E501,W503,E203
return-status-codes = True
pre-commit-hook = False

[omelette]
recipe = collective.recipe.omelette
eggs = ${instance:eggs}

[test]
recipe = collective.xmltestreport
eggs = ${instance:eggs}
defaults = ['-s', 'customer.policy', '--auto-color', '--auto-progress']
environment = environment

[robot]
recipe = zc.recipe.egg
eggs =
${test:eggs}
plone.app.robotframework[debug,reload]

[robot-server]
recipe = zc.recipe.egg
eggs =
${instance:eggs}
collective.MockMailHost
plone.app.robotframework
robotframework-debuglibrary
robotframework-requests
robotframework-react
robotframework-seleniumlibrary
robotframework-selenium2library
robotframework-webpack
scripts =
robot-server
pybot

[scripts]
recipe = zc.recipe.egg
eggs =
zest.releaser
black
i18ndude

[environment]
ROBOT_SELENIUM2LIBRARY_RUN_ON_FAILURE = Capture page screenshot and log source
`````

Any clue or pointer what we are doing wrong here? That problem popped up yesterday even though all our dependencies are pinned. cc @jensens @sneridagh

[mauritsvanrees]

See mostly my comment plus a few comments below it.

Summary: on any Plone 5.2 version, if you use zodb-temporary-storage = off, either the instance fails at startup, or you get a warning. That depends on the versions of Products.Sessions and Products.TemporaryFolder. 5.2.4 has good pins for these. On existing instances, you will need to manually remove the temp_folder object in the root of the ZMI, but I think with those versions it will get recreated anyway. I am working from memory here.

On Plone 6 there are newer versions that do not create the temp_folder object, and an upgrade step that removes it.

So basically: on Plone 5.2 you should stick to zodb-temporary-storage = on.

[mauritsvanrees]

The underlying issue that made tempstorage unreliable, was fixed and version 5.2 has been released today. See also https://github.com/zopefoundation/Zope/issues/985
The new version should be fine to use in Plone 5.2 and 6.0.

The changes that I already made earlier for 6.0 should still be fine. Especially https://github.com/plone/plone.app.upgrade/pull/252 removes the temp_folder from the ZMI, but only does this when it is broken: when any Products.TemporaryFolder version is in the eggs, this upgrade step will let it stay.