Server: Web upload should probably ignore OSX pointer files

[RNCTX]

Steps to reproduce

1. Drag and drop a folder from an OSX client
2. cd /path/to/www/nextcloud/data/user/folder
3. find . '*.ext'

Expected behaviour

Nextcloud should probably ignore hidden metadata files that OSX puts all over its file system.

Actual behaviour

For every file uploaded in this manner from an OSX client, there will be a hidden metadata file beginning in ._

(ex myfile.pdf will have another named ._myfile.pdf)

Server configuration

Operating system:
FreeBSD 11.1-STABLE
Web server:
Nginx 1.12
Database:
postgres 9.5
PHP version:
7.1
Nextcloud version: (see Nextcloud admin page)
13.0.2
Updated from an older Nextcloud/ownCloud or fresh install:
fresh install
Where did you install Nextcloud from:
official site zip file

Client configuration

Browser:
Chrome v66.0.3359.181
Operating system:
OSX

I can't find the document on an Apple domain, but here's an edu user still hosting the Apple document from way back when, explaining OSX's file metadata behavior (from 2001). I can confirm that as of the current version of OSX this has not changed...

http://barbie.uta.edu/~jli/Resources/Resource%20Provisoning&Performance%20Evaluation/81.pdf

On page 182, in "The Finder and File Operations":

When the Finder copies or moves a file, it uses the richest model available, given the
formats of the source and destination volumes. The formats that are most significant
in these kinds of operations are HFS+ (or HFS) and UFS. These operations
particularly affect the representation of the HFS and HFS+ resource fork and the
Finder attributes, especially the type and creator codes.

As one might expect, the Finder preserves the resource fork and Finder attributes of
an HFS+ file “as is” when it copies the file to an HFS+ (or HFS) volume. The more
interesting case, however, is when it copies an HFS+ file to a UFS volume. When this
happens, the Finder splits out the information that is not in the data fork
(particularly the type and creator codes) and writes this information to a hidden file
in the same directory location as the copied file. This hidden file has the same name
as the UFS file, except that it has a “dot-underscore” prefix. Thus, if you have an
HFS+ file named MyMug.jpeg, when you copy it to a UFS volume, there will be a file
named ._MyMug.jpeg in the same location.

When the Finder copies a UFS file to an HFS or HFS+ volume, it looks for the hidden
“dot-underscore” file. If one exists, it creates an HFS+ (or HFS) file reintegrating the
information in the hidden file into the file’s resource fork and Finder attributes. If
the hidden file does not exist, the copied file has no resource fork.

I get that this is a really stupid design on Apple's part to litter users' file systems with gazillions of hidden files, and it's not the fault of Nextcloud, but it would be nice for batch processing apps/functions if Nextcloud could ignore, or have the option to ignore, these OSX metadata files (.DS_Store being another notorious one, that everyone knows about) from batch uploads such as folder drag/drop operations.

[tflidd]

If you use the nextcloud client, https://github.com/nextcloud/client/issues is the better place for a report. If you use it as a webdav-network-drive, then it should stay.

[RNCTX]

The report is in reference to the web interface (hence listing my Chrome version). I do not use the desktop client(s).

[MorrisJobke]

cc @nextcloud/javascript

IMO this makes a lot of sense

[RNCTX]

Rationale from my POV as an OSX desktop user and *nix server user, I know some of this is stating the obvious but if I were to have to argue this to a jury in a courtroom ;) :

1) If I wanted to retain OSX's metadata files (for a proper backup of a whole or part of a filesystem) I would use another tool to ensure that hidden files/folders are transferred such as tar archives, git, rsync, OSX's integral backup tool, etc. I have no reason to expect that a browser upload accesses hidden files outside of the directories that belong to the browser application.

2) If a user knows enough about *nix filesystems to know that hidden files begin with a dot, they should rightfully assume that a web browser function does not expose hidden files on their system. All browsers on OSX ignore a user's request to upload their entire home folder from OSX via a browser, because that would include their .ssh folders, for example.

3) I think the number 2 example is a sort of 'scorched earth' policy, in that the browser refuses to perform the user's request, but a default-off option to ignore dot-hidden files from uploads seems like a wise change, in keeping with common practices of handling dot-hidden *nix files. If a user actively seeks out the setting which is preventing him/her from uploading dot-hidden files and disables that setting despite warnings to the contrary, then the responsibility belongs to the user.

4) As mentioned in the initial report, other utilities which batch-process uploads that want to list and sort files uploaded to nextcloud will have to have another layer of error-handling, because *nix 'find' for example will locate the hidden files. That's how I stumbled across this, I was writing a shell script to look at PDF files uploaded to nextcloud, and for those without text, process them with an OCR library, but my script was finding OSX metadata files as well. I had dragged and dropped a folder containing the PDF files from a Linux server to Nextcloud mounted via OSX's Finder in the middle (on my laptop).

5) Remote filesystem mounting/serving tools such as Samba necessarily have options for the administrator or user to ignore or not ignore dot-hidden files, because users might want to, for example, run applications remotely that need such functionality. But such tools are distinctively not web browser based and therefore do not rely on web browser policies. The user has a right to assume proper minimum security behavior from a web browser in this day and age, I think, unless they explicitly configure the system otherwise.

6) I understand that services such as Dropbox and Google Drive, and applications such as Nextcloud, occupy a grey area in number 5, in that they are supposed to sync things and have browser accessibility as an option, so with that in mind if it were up to me, a broad policy of "disable by default, enable with explicit warning" makes sense in this context.

[skjnldsv]

@MorrisJobke should we create a list to ignore like our desktop client?

[RNCTX]

Update to this:

As you guys may or may not have noticed if you've worked on this (I have since I'm utilizing WebDAV in an unrelated project...), there is no way to stop OSX from doing this.

So this becomes a bit of a conundrum, upon further investigation. If you stop the OSX client from putting the dot-underscore files in there or delete the files after it copies them, it will (likely) keep trying to put them back. I'm not sure if iOS does the same thing or not.

Here's the info-level output of a Rails app that I'm working with. In trying to get an OSX client talking to the Rails implementation of WebDAV, this is what happens immediately after the OSX client authenticates:

I, [2018-08-27T17:42:31.951648 #29134]  INFO -- : Started LOCK "/webdav/.DS_Store" for 192.168.1.1 at 2018-08-27 17:42:31 -0500
I, [2018-08-27T17:42:31.956540 #36034]  INFO -- : Started PROPFIND "/webdav/._Beechcraft" for 192.168.1.1 at 2018-08-27 17:42:31 -0500
I, [2018-08-27T17:42:31.968143 #29134]  INFO -- : Started PROPFIND "/webdav/.DS_Store" for 192.168.1.1 at 2018-08-27 17:42:31 -0500
I, [2018-08-27T17:42:31.977465 #29134]  INFO -- : Started PROPFIND "/webdav/._Piper" for 192.168.1.1 at 2018-08-27 17:42:31 -0500
I, [2018-08-27T17:42:31.985922 #29134]  INFO -- : Started PROPFIND "/webdav/._Appareo" for 192.168.1.1 at 2018-08-27 17:42:31 -0500
I, [2018-08-27T17:42:31.997406 #29134]  INFO -- : Started PROPFIND "/webdav/._Garmin" for 192.168.1.1 at 2018-08-27 17:42:31 -0500
I, [2018-08-27T17:42:32.012099 #29134]  INFO -- : Started PROPFIND "/webdav/._Continental" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.037552 #29134]  INFO -- : Started PROPFIND "/webdav/" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.075780 #29134]  INFO -- : Started PROPFIND "/webdav/._Narco" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.085319 #29134]  INFO -- : Started PROPFIND "/webdav/._King" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.093756 #29134]  INFO -- : Started PROPFIND "/webdav/._." for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.102617 #29134]  INFO -- : Started PROPFIND "/webdav/._Diamond%2040" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.111850 #29134]  INFO -- : Started PROPFIND "/webdav/.DS_Store" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.121465 #29134]  INFO -- : Started PROPFIND "/webdav/._PS%20Engineering" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.135168 #29134]  INFO -- : Started PROPFIND "/webdav/._Interav" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.149347 #29134]  INFO -- : Started PROPFIND "/webdav/" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.197219 #29134]  INFO -- : Started PROPFIND "/webdav/._E.I." for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.209227 #29134]  INFO -- : Started PROPFIND "/webdav/" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.225705 #29134]  INFO -- : Started PROPFIND "/webdav/._Trig" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.239484 #29134]  INFO -- : Started PROPFIND "/webdav/._Grumman" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.255156 #29134]  INFO -- : Started PROPFIND "/webdav/" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.309564 #29134]  INFO -- : Started PROPFIND "/webdav/._Plane%20Power" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.318652 #29134]  INFO -- : Started PROPFIND "/webdav/._Navworx" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.330345 #29134]  INFO -- : Started PROPFIND "/webdav/._Century" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.343673 #29134]  INFO -- : Started PROPFIND "/webdav/._Cessna" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.356480 #29134]  INFO -- : Started PROPFIND "/webdav/._Transcal" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.372014 #29134]  INFO -- : Started PROPFIND "/webdav/._Aspen%20Avionics" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.387780 #29134]  INFO -- : Started PROPFIND "/webdav/._Continental%20Aero" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.402557 #29134]  INFO -- : Started PROPFIND "/webdav/._Manuals" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.416025 #29134]  INFO -- : Started PROPFIND "/webdav/._S-TEC" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.431174 #29134]  INFO -- : Started PROPFIND "/webdav/._Stormscope" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.445125 #29134]  INFO -- : Started PROPFIND "/webdav/._Lycoming" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.460110 #29134]  INFO -- : Started PROPFIND "/webdav/._JPInstruments" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.474989 #29134]  INFO -- : Started PROPFIND "/webdav/._Sandel" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.490162 #29134]  INFO -- : Started PROPFIND "/webdav/._Commander" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.505189 #29134]  INFO -- : Started PROPFIND "/webdav/._Mooney" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.519377 #29134]  INFO -- : Started PROPFIND "/webdav/._Cirrus" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.533269 #29134]  INFO -- : Started PROPFIND "/webdav/._Aircraft%20Inspection" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.549568 #29134]  INFO -- : Started PROPFIND "/webdav/._Aviation%20Mechanic" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.563987 #29134]  INFO -- : Started PROPFIND "/webdav/._Avidyne" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.582202 #29134]  INFO -- : Started PROPFIND "/webdav/Navworx/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.595886 #29134]  INFO -- : Started PROPFIND "/webdav/Cessna/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.610940 #29134]  INFO -- : Started PROPFIND "/webdav/Aspen%20Avionics/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.624925 #29134]  INFO -- : Started PROPFIND "/webdav/Pratt%20&%20Whitney/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.639981 #29134]  INFO -- : Started PROPFIND "/webdav/Pratt%20&%20Whitney/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.641254 #36034]  INFO -- : Started PROPFIND "/webdav/Beechcraft/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.655243 #29134]  INFO -- : Started PROPFIND "/webdav/Piper/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.670972 #29134]  INFO -- : Started PROPFIND "/webdav/Piper/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.671683 #36034]  INFO -- : Started PROPFIND "/webdav/Appareo/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.686476 #29134]  INFO -- : Started PROPFIND "/webdav/Garmin/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.700566 #29134]  INFO -- : Started PROPFIND "/webdav/Continental/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.716376 #29134]  INFO -- : Started PROPFIND "/webdav/Narco/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.731656 #29134]  INFO -- : Started PROPFIND "/webdav/King/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.746442 #29134]  INFO -- : Started PROPFIND "/webdav/Diamond%2040/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.760734 #29134]  INFO -- : Started PROPFIND "/webdav/PS%20Engineering/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.775552 #29134]  INFO -- : Started PROPFIND "/webdav/Interav/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.789874 #29134]  INFO -- : Started PROPFIND "/webdav/E.I./.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.803966 #29134]  INFO -- : Started PROPFIND "/webdav/Trig/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.818540 #29134]  INFO -- : Started PROPFIND "/webdav/Grumman/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.833437 #29134]  INFO -- : Started PROPFIND "/webdav/Plane%20Power/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.849281 #29134]  INFO -- : Started PROPFIND "/webdav/Century/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.862719 #29134]  INFO -- : Started PROPFIND "/webdav/Transcal/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.876652 #29134]  INFO -- : Started PROPFIND "/webdav/Continental%20Aero/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.876869 #36034]  INFO -- : Started PROPFIND "/webdav/Transcal/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.891197 #29134]  INFO -- : Started PROPFIND "/webdav/Manuals/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.905952 #29134]  INFO -- : Started PROPFIND "/webdav/S-TEC/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.921062 #29134]  INFO -- : Started PROPFIND "/webdav/Stormscope/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.937065 #29134]  INFO -- : Started PROPFIND "/webdav/Lycoming/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.938111 #36034]  INFO -- : Started PROPFIND "/webdav/Stormscope/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.952128 #29134]  INFO -- : Started PROPFIND "/webdav/JPInstruments/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.967592 #29134]  INFO -- : Started PROPFIND "/webdav/Sandel/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.982307 #29134]  INFO -- : Started PROPFIND "/webdav/Commander/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.997923 #29134]  INFO -- : Started PROPFIND "/webdav/Commander/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:32.998890 #36034]  INFO -- : Started PROPFIND "/webdav/Mooney/.localized" for 192.168.1.1 at 2018-08-27 17:42:32 -0500
I, [2018-08-27T17:42:33.013428 #29134]  INFO -- : Started PROPFIND "/webdav/Cirrus/.localized" for 192.168.1.1 at 2018-08-27 17:42:33 -0500
I, [2018-08-27T17:42:33.028498 #29134]  INFO -- : Started PROPFIND "/webdav/Aircraft%20Inspection/.localized" for 192.168.1.1 at 2018-08-27 17:42:33 -0500
I, [2018-08-27T17:42:33.043706 #29134]  INFO -- : Started PROPFIND "/webdav/Aircraft%20Inspection/.localized" for 192.168.1.1 at 2018-08-27 17:42:33 -0500
I, [2018-08-27T17:42:33.044287 #36034]  INFO -- : Started PROPFIND "/webdav/Aviation%20Mechanic/.localized" for 192.168.1.1 at 2018-08-27 17:42:33 -0500
I, [2018-08-27T17:42:33.060315 #29134]  INFO -- : Started PROPFIND "/webdav/Avidyne/.localized" for 192.168.1.1 at 2018-08-27 17:42:33 -0500
I, [2018-08-27T17:42:33.075344 #29134]  INFO -- : Started PROPFIND "/webdav/Avidyne/.localized" for 192.168.1.1 at 2018-08-27 17:42:33 -0500
I, [2018-08-27T17:42:33.104173 #29134]  INFO -- : Started PROPFIND "/webdav/.localized" for 192.168.1.1 at 2018-08-27 17:42:33 -0500
I, [2018-08-27T17:42:37.015203 #29134]  INFO -- : Started PROPFIND "/webdav/.DS_Store" for 192.168.1.1 at 2018-08-27 17:42:37 -0500
I, [2018-08-27T17:42:37.030826 #29134]  INFO -- : Started PROPFIND "/webdav/._.DS_Store" for 192.168.1.1 at 2018-08-27 17:42:37 -0500
I, [2018-08-27T17:42:37.042033 #29134]  INFO -- : Started GET "/webdav/._.DS_Store" for 192.168.1.1 at 2018-08-27 17:42:37 -0500
I, [2018-08-27T17:42:37.053449 #29134]  INFO -- : Started LOCK "/webdav/.DS_Store" for 192.168.1.1 at 2018-08-27 17:42:37 -0500
I, [2018-08-27T17:42:37.069311 #29134]  INFO -- : Started GET "/webdav/.DS_Store" for 192.168.1.1 at 2018-08-27 17:42:37 -0500
I, [2018-08-27T17:42:37.081103 #29134]  INFO -- : Started PUT "/webdav/.DS_Store" for 192.168.1.1 at 2018-08-27 17:42:37 -0500
I, [2018-08-27T17:42:37.091523 #29134]  INFO -- : Started UNLOCK "/webdav/.DS_Store" for 192.168.1.1 at 2018-08-27 17:42:37 -0500

In my case there are other issues because of Nginx not having a complete implementation of WebDAV so I don't have further info yet, but my case is a good one for seeing what happens when OSX fails to do what it wants to do, because my setup with Nginx was broken in regard to uploads.

What will OSX do if it fails to find them and fails to put them there? Good question. I'll know tomorrow or the day after, I have to replace my setup with Apache to get WebDAV working in my Rails example. I do know at this point that it will fail silently on those metadata files, but my broken Nginx setup would cause Passenger to crash immediately after the failed upload so I couldn't test further.

There was a comment on Stackoverflow from the SabreDAV author in regard to this that I stumbled upon but can't put my finger on right this minute, I'll link his solution to this as well if I can find it again. The gist of his solution was that there was no way to stop OSX from doing this, so he implemented functions server-side that ignored these files within SabreDAV, and then deleted them with cron periodically.