Libertem: Basic File Browser

[sk1p]

For opening local HDF5 / RAW files

[sk1p]

Could additionally accept Drag&Drop events - https://developer.mozilla.org/en-US/docs/Web/API/HTML_Drag_and_Drop_API/File_drag_and_drop

[uellue]

Drag and drop sounds awesome! We should make sure that the entire backend is local and display a helpful message if it is not. I'm quite sure someone will try this with a remote cluster...

[sk1p]

Yeah, otherwise it could offer to upload & ingest the dropped file, though I don't know how efficient it is to do the upload via the browser. Could actually work...

[uellue]

Good idea to offer upload! With a confirmation step because it might not be what the user wants to do, for example if the same network file system or cloud folder is mounted on both client and server.

[sk1p]

Yeah, also the binary formats may need some parameters to be read...

So the flow would be:

1) Drop file
2) Get a parameters dialog
3) If the file is remote, start uploading (with an indicator somewhere on the screen - normal work can continue)
4) If the file is not remote, or on completion of upload, open file as Dataset

The "normal" (non-DnD) flow should be the same actually - you first select a file (from local file system or from HDFS), then you get the parameter dialog, and then it is opened. This enables us to make some things more user friendly, for example we could pre-fill some form fields etc. For EMPAD we can at least guess the scan dimensions from the filesize, and provide information for form input validation, which is a bit awkward if the file name is also part of the same form.

I'll create a separate ticket for uploading via DnD - I think it fits for the 0.2 milestone, do you agree?

[sk1p]

Actually DnD for direct reading (without 'uploading') will be limited, as the fullPath property is not universally supported.

[sk1p]

For the client, we should use react-window to render the list of files, to support directories with many files.

Need to think about how to properly open MIB files - idea: instead of specifying a glob pattern, just select the *.hdr file, use its 'basename' to construct a glob: default.hdr -> default*.mib

[sk1p]

There are some security considerations concerning the file browser (or even without the file browser: loading from arbitraty paths) and multi-user environments - LiberTEM always has access to all files of the user it is running as - including sensitive information like passwords or SSH keys, and other users can access that LiberTEM instance.

We should think about this a bit more, and at least introduce a cmdline switch to whitelist access to certain paths, where the user expects to load data from, or generate a token which the user has to include when he first accesses the UI.

[uellue]

Hm, how does, for example, Nextcloud handle authentication? Is there a "best practice" for a system like LiberTEM that gives a web-based interface for local files? Can we assume that LiberTEM is always running as the Unix user that belongs to the "web user" or might there be web users that don't have a local Unix account?I'm just a bit worried because in particular on a cluster it gets messy IMO if we can't rely on the operating system to manage access. We should probably not roll our own solution but rely on good third party code. Is there a code base we could use?

[sk1p]

Agreed on not rolling our own.

Things like Nextcloud require you to authenticate against some kind of user database (either built-in or some kind of external database like LDAP, OAuth, or even against other network services like IMAP, SMB, FTP, ...). That part should not be problematic.

For authorization, LiberTEM can't be compared to Nextcloud, as Nextcloud doesn't access arbitrary files, it just stores its files in a data directory, with its own ACL database on top.

I would distinguish the following deployment scenarios that have different security requirements:

1) Single-user system: LiberTEM can run as the login user and thus access all files of the user, and should only listen on the loopback interface. No need for authentication or authorization, as it would add complexity without a real threat.

2) Shared computer, listening on loopback: LiberTEM should run as a separate (non-privileged) user, and can only access users' files if they allow it via standard file system measures. As long as LiberTEM only listens on a loopback interface, no trust boundary is crossed: anyone who can locally access or SSH into the system could access those files directly anyways. This would not work if users need to access confidential data sets.

3) Shared computer, listening on public address: authentication (and SSL) becomes necessary. If any of the data is confidential, further access checks (authorization) become necessary, maybe including a mapping of "web user" to UNIX users. This can become messy. Imagine spawning I/O processes as the respective UNIX user which then somehow communicate with the worker processes. Easiest way: fd passing via UNIX domain socket. But that would not work for all file formats (case in point: HDF5 does not allow opening via fd). Or there could be a layer above, that handles authentication and spawns a whole LiberTEM server for each user, which of course costs more memory, but would delegate all file access control to the OS.

4) Cluster with HDFS: Authentication and Authorization via Kerberos - users authenticate with their LDAP credentials, authorization is integrated with HDFS. Ingestion should allow to set an umask with a secure default (077 to disallow all other users access by default). I think most work would be actually the initial HDFS setup and making sure the hdfs3 python lib works well with kerberos.

As for standards/best practice: most web-based systems are limited to a white-list of directories.

I think a good role model is jupyterhub (see also 1, 2) - they actually use the "spawn server per user" model. And when you are using jupyter notebooks without the hub, they use a token:

http://localhost:8888/?token=XXXX

Need to think and research more about this.

[davidelbert]

I'm new to these LiberTEM discussions, but want to suggest checking out Globus authentication. It provides access to SSO for academic and lab users as well as OpenID (Google authentication), and InCommon. It would also make it easy to tap into a growing materials data infrastructure where Globus is the de facto standard for large file transfer and offers storage solution. There are lots of resources to learn more, but one place to start would be the Globus docs: https://docs.globus.org/authorization-authentication-guide/

[sk1p]

@davidelbert thank you for joining the discussion! Globus looks interesting, from a quick glance it looks similar to the B2ACCESS/B2SHARE/B2DROP services provided by EUDAT, right? I'd love to learn more about that, do you have first-hand experience using this system?

[sk1p]

Let's continue the auth* discussion on #67.

I committed a first version of the file browser, it works but still has a few glitches and can be improved UI-wise. It only shows the local file system, it needs to be enhanced to also support browsing HFDS file systems.

Especially under Chrome on Windows, the file browser doesn't show files until first scrolling. Chome on Linux sometimes only shows some files, until scrolling. This appears to be an issue with react-window, investigating.

[sk1p]

"disappearing files" bug filed upstream: https://github.com/bvaughn/react-window/issues/45

[uellue]

4ca7310 fixed the scrolling issue for me. Thx @sk1p !