Libertem: Don't litter cwd with dask-worker-space

[sk1p]

It is created by dask.distributed wherever you start LiberTEM - it should instead be created in a fixed location (~/.cache? Mac OS X / Win?)

[uellue]

@sk1p what is in that folder?

1. Should it persist between sessions?
2. Might it contain temporary data of significant size that we should delete in case LiberTEM doesn't exit cleanly?
3. Does it contain things like lock files that are relevant for concurrent sessions or that might mess things up if they linger in case LiberTEM didn't exit cleanly?
4. Can two instances share the same folder?
5. Might data accumulate over time, like log files?
6. Is the IO performance or latency of the underlying file system performance-critical somehow?

[uellue]

Did some research:

• We can have large temporary data here
• SIGKILL will make it linger
• The workers will try to get a lock for their directory

https://github.com/dask/distributed/pull/1543
https://stackoverflow.com/questions/48657374/what-is-the-default-directory-where-dask-workers-store-results-or-files
https://distributed.readthedocs.io/en/latest/worker.html#spill-data-to-disk
https://github.com/dask/distributed/issues/1540
https://github.com/dask/distributed/issues/2113

Is this a different directory?

https://github.com/dask/dask/issues/1659

Deactivate dumping data there. I guess we should do that because we read from files anyway and it should never be necessary, if I understand correctly. I'd rather have workers die than clutter disk space.

http://dask.pydata.org/en/latest/setup/hpc.html#no-local-storage

So that would mean:

1. _Should it persist between sessions?_ No.
2. _Might it contain temporary data of significant size that we should delete in case LiberTEM doesn't exit cleanly?_ YES -- but not for LiberTEM PROVIDED everything works as it should!
3. _Does it contain things like lock files that are relevant for concurrent sessions or that might mess things up if they linger in case LiberTEM didn't exit cleanly?_ No, since workers only lock their own dir. But the file system needs to support locking.
4. _Can two instances share the same folder?_ Yes, it seems, since each worker creates their own unique folder.
5. _Might data accumulate over time, like log files?_ Yes, leftover dirs accumulate when the worker (or scheduler?) dies unexpectedly.
6. _Is the IO performance or latency of the underlying file system performance-critical somehow?_ YES, if it is used for large intermediate data. Should not be relevant for LiberTEM IF everything works as expected.

What about using %TEMP%\libertem\dask on Windows? That expands to C:\Users\<user>\AppData\Local\Temp\libertem\dask on my system (Windows 10).

[uellue]

On Unix this sounds like a job for a private subdirectory in /tmp/, I guess?

http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s18.html

[uellue]

Or maybe this is relevant? https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

[uellue]

I'm not sure if the current behavior of dask to just use the working directory to store large intermediate data is what the user would expect. IMO it should by default just use a normal temporary directory for locking or small data and throw an error if it runs out of memory. The "out of memory" error message could contain a helpful suggestion to specify a folder for large intermediary data IF that is what the user wants.

[robbmcleod]

Probably what you want to use here is the tempfile module:

https://docs.python.org/3/library/tempfile.html#tempfile.TemporaryDirectory

[sk1p]

Thanks @robbmcleod - I wasn't aware of the TemporaryDirectory class. It appears to be a nice, modern approach to mkdtemp! Temporary storage should also work nicely (always supports locking AFAIK), at least for our case of no spilling.

[sk1p]

There is also https://pypi.org/project/appdirs/ - specifically user_cache_dir

[samcmill]

+1

This is also an issue for read-only container images.

[sk1p]

Closed via #345