Core: .ocTransferIDXXXXX.part makes filename too long

Created on 8 Jul 2016 · 36Comments · Source: owncloud/core

Just ran into that error on my Server, my exact Filename (encrypted by boxcryptor) is "倐哤幇岓奣划圵囩刜倃傤墑倩墌峂囱存底堝堁形屜孭啜屬囀凝匥媵忨噪厪婪媞妲匳墩密嵡寕征嫴啈宧吴厁宊埧姠偁儽忺入哄寘嚂屧剚室堙嶞初崵峋凗堡徟塮傳圔媀吽嶌孲執嬊夾埬塝䀋.bc" which is 245 bytes long. after adding the .ocTransfer suffix it has 272 bytes…

ssh@server:/www/htdocs$ echo 倐哤幇岓奣划圵囩刜倃傤墑倩墌峂囱存底堝堁形屜孭啜屬囀凝匥媵忨噪厪婪媞妲匳墩密嵡寕征嫴啈宧吴厁宊埧姠偁儽忺入哄寘嚂屧剚室堙嶞初崵峋凗堡徟塮傳圔媀吽嶌孲執嬊夾埬塝䀋.bc | wc -c
245
ssh@server:/www/htdocs$ echo 倐哤幇岓奣划圵囩刜倃傤墑倩墌峂囱存底堝堁形屜孭啜屬囀凝匥媵忨噪厪婪媞妲匳墩密嵡寕征嫴啈宧吴厁宊埧姠偁儽忺入哄寘嚂屧剚室堙嶞初崵峋凗堡徟塮傳圔媀吽嶌孲執嬊夾埬塝䀋.bc.ocTransferId371634213.part | wc -c
272

that suffix should be kept in mind when limiting uploads to OC - or even better: a shorter or no suffix should be used.

ownCloud 9.0.3

Bug blue-ticket filesystem sev2-high

Source

juliangieseke

All 36 comments

OC Logs:

Fatal webdav Exception: {"Message":"HTTP\/1.1 500 Could not write file contents","Exception":"Sabre\DAV\Exception","Code":0,"Trace":"#0 \/www\/htdocs\…/owncloud-9.0.3\/apps\/dav\/lib\/connector\/sabre\/directory.php(134): OCA\DAV\Connector\Sabre\File->put(Resource id #338)\n#1 \/www\/htdocs\…/owncloud-9.0.3\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(1036): OCA\DAV\Connector\Sabre\Directory->createFile('\xE5\x80\x90\xE5\x93\xA4\xE5\xB9\x87\xE5\xB2\x93\xE5\xA5\xA3...', Resource id #338)\n#2 \/www\/htdocs\…/owncloud-9.0.3\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(523): Sabre\DAV\Server->createFile('Boxcryptor\/Doku...', Resource id #338, NULL)\n#3 [internal function]: Sabre\DAV\CorePlugin->httpPut(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))\n#4 \/www\/htdocs\…/owncloud-9.0.3\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#5 \/www\/htdocs\…/owncloud-9.0.3\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(459): Sabre\Event\EventEmitter->emit('method:PUT', Array)\n#6 \/www\/htdocs\…/owncloud-9.0.3\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(248): Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))\n#7 \/www\/htdocs\…/owncloud-9.0.3\/apps\/dav\/appinfo\/v1\/webdav.php(55): Sabre\DAV\Server->exec()\n#8 \/www\/htdocs\…/owncloud-9.0.3\/remote.php(138): require_once('\/www\/htdocs\/w01...')\n#9 {main}","File":"\/www\/htdocs\…/owncloud-9.0.3\/apps\/dav\/lib\/connector\/sabre\/file.php","Line":130,"User":"julian.gieseke"} 2016-07-08T18:53:01+00:00
Error webdav \OC\Files\Filesystem::fopen() failed 2016-07-08T18:53:01+00:00
Error PHP fopen(/www/htdocs/…/倐哲奄姢冟嶑婥寒坄寜䂄/倐听妃堲咛嚤嫓䀏/倐咫咫埩啹啡䀅/倐后困尦帟劻䃡/倐哤幇岓奣划圵囩刜倃傤墑倩墌峂囱存底堝堁形屜孭啜屬囀凝匥媵忨噪厪婪媞妲匳墩密嵡寕征嫴啈宧吴厁宊埧姠偁儽忺入哄寘嚂屧剚室堙嶞初崵峋凗堡徟塮傳圔媀吽嶌孲執嬊夾埬塝䀋.bc.ocTransferId371634213.part): failed to open stream: File name too long at /www/htdocs/…/owncloud-9.0.3/lib/private/files/storage/local.php#261 2016-07-08T18:53:01+00:00

juliangieseke on 8 Jul 2016

Error PHP fopen(/www/htdocs/…/倐哲奄姢冟嶑婥寒坄寜䂄/倐听妃堲咛嚤嫓䀏/倐咫咫埩啹啡䀅/倐后困尦帟劻䃡/倐哤幇岓奣划圵囩刜倃傤墑倩墌峂囱存底堝堁形屜孭啜屬囀凝匥媵忨噪厪婪媞妲匳墩密嵡寕征嫴啈宧吴厁宊埧姠偁儽忺入哄寘嚂屧剚室堙嶞初崵峋凗堡徟塮傳圔媀吽嶌孲執嬊夾埬塝䀋.bc.ocTransferId371634213.part): failed to open stream: File name too long

This error message isn't generated in ownCloud, this means that the fopen call itself caused that issue.

Please use the issue template, would be good to know what system and filesystem you are running on that would cause such errors. Is that an external storage ?

https://raw.githubusercontent.com/owncloud/core/master/issue_template.md

PVince81 on 11 Jul 2016

I guess maybe part files don't even need to have the real filename in them as long as the transfer id is there for the final rename.

@owncloud/filesystem

PVince81 on 11 Jul 2016

Steps to reproduce

create File with Name longer then 229(maybe 228) bytes

Expected behaviour

file should be uploaded

Actual behaviour

php error message above

Server configuration

Operating system:
ubuntu (shared hosting)
Web server:
shared hosting, apache 2.x
Database:
mysql (5.x)
PHP version:
7.0.6
ownCloud version: (see ownCloud admin page)
9.0.3
Updated from an older ownCloud or fresh install:
updated from 9.0.2
Where did you install ownCloud from:
owncloud.org website
Signing status (ownCloud 9.0 and above):
passed.

List of activated apps:

Enabled:
  - activity: 2.2.1
  - announcementcenter: 1.1.2
  - calendar: 1.2.2
  - comments: 0.2
  - contacts: 1.3.1.0
  - dav: 0.1.6
  - federatedfilesharing: 0.1.0
  - federation: 0.0.4
  - files: 1.4.4
  - files_pdfviewer: 0.8.1
  - files_sharing: 0.9.1
  - files_texteditor: 2.1
  - files_trashbin: 0.8.0
  - files_versions: 1.2.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 14.5.0
  - notifications: 0.2.3
  - provisioning_api: 0.4.1
  - systemtags: 0.2
  - templateeditor: 0.1
  - updatenotification: 0.1.0
Disabled:
  - encryption
  - external
  - files_antivirus
  - files_external
  - mail
  - user_external
  - user_ldap

The content of config/config.php:

{
    "system": {
        "instanceid": "ocgbgmawcg1g",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "mydomain"
        ],
        "datadirectory": "\/www\/htdocs\/mydomain\/data",
        "overwrite.cli.url": "https:\/\/mydomain",
        "dbtype": "mysql",
        "version": "9.0.3.2",
        "dbname": "mydb",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "default_language": "de_DE",
        "tempdirectory": "\/www\/htdocs\/mydomain\/tmp",
        "mail_from_address": "technik",
        "mail_smtpmode": "smtp",
        "mail_domain": "mydomain",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "log_rotate_size": 104857600,
        "mail_smtpsecure": "tls",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "mydomain",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "loglevel": 2
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
nope
Are you using encryption: yes/no
nope
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
nope

Client configuration

Browser:
OS X Desktop client?!
Operating system:
OS X

Logs

Web server error log

shared hosting

ownCloud log (data/owncloud.log)

see above

juliangieseke on 11 Jul 2016

Shared hosting, ok. Maybe they added a limit to file name lengths.

PVince81 on 11 Jul 2016

I dont think its a Shared-Hosting-Specific problem, ext4 has 255bytes as max filename length.

juliangieseke on 11 Jul 2016

@DeepDiver1975 so we should limit file names to less than 255 due to suffixes.

PVince81 on 11 Jul 2016

Maybe removing the original filename from .part files is a better idea… limiting filenames to <255 bytes could lead to the same problem: files generated by boxcryptor (and other encryption tools) cant be uploaded.

juliangieseke on 11 Jul 2016

@dragotin @guruz something for you to have a look at ?

PVince81 on 11 Jul 2016

also owncloud/enterprise#1366

Note that with the new chunking this problem is fixed anyway.

dragotin on 11 Jul 2016

A workaround might be:

If the filename of the chunk-file incl. path is bigger than 250 chars, the actual file component is checked if it is long and can be shortened to fit the chunked-<upload-id>-<count>-<overall-no> that comes with the chunk file. If that works, the client will do that and after the file was uploaded it remembers to send a rename.

I am actually unsure as a) that is hacky, b) it is fixed with the new chunking anyway, c) its still a big change.

dragotin on 11 Jul 2016

@dragotin I think the fix doesn't require to affect whatever the client is sending.

The fix only for how the part file and chunk file are stored on-disk. The name already doesn't match 100% what the client sends. So we could maybe md5 the filename there.

PVince81 on 11 Jul 2016

👍1

Might be slightly related: https://github.com/owncloud/core/issues/25214

Strange thing that boxcryptor generates names containing traditional chinese characters.

PVince81 on 11 Jul 2016

https://dragotin.wordpress.com/tag/chunked-upload/

cdamken on 14 Jul 2016

I suggest to simply md5 the base name of the chunk file name on disk on the server side.
Might be just a matter of changing some code in the FileChunking class.

@guruz are you still on this ?

PVince81 on 22 Dec 2016

@IljaN please have a look. The chunking code is here: https://github.com/owncloud/core/blob/v9.1.3/apps/dav/lib/Connector/Sabre/File.php#L395 and the class is here: https://github.com/owncloud/core/blob/v9.1.3/lib/private/legacy/filechunking.php#L79.

Let me know if you need some pointers regarding how to execute this scenario and also how the whole thing works.

PVince81 on 9 Jan 2017

Looks like the encryption code requires the file name prefix to be the same, so that's why we never thought of changing the name: https://github.com/owncloud/core/pull/26978#issuecomment-275066787

PVince81 on 25 Jan 2017

Not fixable short term due to encryption blocker, needs scheduling: https://github.com/owncloud/core/issues/27142

Moving to backlog

PVince81 on 13 Apr 2017

@PVince81 Is this still relevant with new chunking?

guruz on 19 Apr 2017

Yes, because chunking is what happens before it lands in the final storage as a part file.

There are reasons to keep the part file to be able to achieve an atomic "rename + overwrite to final file name" in one go, so part files won't go away anytime soon.

However as for the part file name, encryption relies on it to find older file keys in case of existing files to overwrite. So we can't rename it.

PVince81 on 19 Apr 2017

@PVince81 Yes, the new chunking also uses part files. But I thought it used shorter file names.
https://github.com/owncloud/client/blob/1ed4eb46f20793b7f0742f4c058f6ce20de9515a/src/libsync/propagateuploadng.cpp#L38 not based on the previous file name.

guruz on 19 Apr 2017

The chunk file name is not related to the part file name.

But even if the part file name can be made shorter, it will still use more characters that will block the max limit.

PVince81 on 19 Apr 2017

OK, @PVince81 is right. The part file is the single-file-that-gets-written-and-then-renamed-to-destination-file-name.
The chunk files are different.

guruz on 19 Apr 2017

Moving to backlog

😿

cdamken on 24 Apr 2017

A new idea from @cdamken which we could use:

generate the part file name on disk using partfile = md5(target Path + name).
encryption code reads this partfile md5 and does a select name from oc_filecache where path_hash=$hash replacing $hash with the part file name
then use this name to resolve key files

PVince81 on 6 Dec 2017

With this https://github.com/owncloud/core/pull/29814 at least no more error logs appears. So far good. But there is one situation I am kind of stuck. While testing with delete operation, which results in moving the file to trashbin. This causes error:
{"reqId":"AhuNsH2nTlknbCZXM1s6","level":3,"time":"2017-12-11T16:18:07+00:00","remoteAddr":"127.0.0.1","user":"admin","app":"PHP","method":"DELETE","url":"\/testing\/remote.php\/webdav\/hhhhhhhhhhhhhhhhhhhhhhhhheeeeeeeeeeeeeeeeeeeeeeeeeeelllllllllllllllllllllllllllllllllllloooooooooooooooooooooooooooooooooooooooooooohhhhhhhhhhhhhhhhhhhhhhhhhoooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwaaaaaaaaaarrrrrrrrreyoubro.txt","message":"rename(\/home\/sujith\/test\/owncloud\/data\/admin\/files\/hhhhhhhhhhhhhhhhhhhhhhhhheeeeeeeeeeeeeeeeeeeeeeeeeeelllllllllllllllllllllllllllllllllllloooooooooooooooooooooooooooooooooooooooooooohhhhhhhhhhhhhhhhhhhhhhhhhoooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwaaaaaaaaaarrrrrrrrreyoubro.txt,\/home\/sujith\/test\/owncloud\/data\/admin\/files_trashbin\/files\/hhhhhhhhhhhhhhhhhhhhhhhhheeeeeeeeeeeeeeeeeeeeeeeeeeelllllllllllllllllllllllllllllllllllloooooooooooooooooooooooooooooooooooooooooooohhhhhhhhhhhhhhhhhhhhhhhhhoooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwaaaaaaaaaarrrrrrrrreyoubro.txt.d1513009038): File name too long at \/home\/sujith\/test\/owncloud\/lib\/private\/Files\/Storage\/Local.php#269"} {"reqId":"AhuNsH2nTlknbCZXM1s6","level":2,"time":"2017-12-11T16:18:07+00:00","remoteAddr":"127.0.0.1","user":"admin","app":"dav","method":"DELETE","url":"\/testing\/remote.php\/webdav\/hhhhhhhhhhhhhhhhhhhhhhhhheeeeeeeeeeeeeeeeeeeeeeeeeeelllllllllllllllllllllllllllllllllllloooooooooooooooooooooooooooooooooooooooooooohhhhhhhhhhhhhhhhhhhhhhhhhoooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwaaaaaaaaaarrrrrrrrreyoubro.txt","message":"Could not get node for path: \"hhhhhhhhhhhhhhhhhhhhhhhhheeeeeeeeeeeeeeeeeeeeeeeeeeelllllllllllllllllllllllllllllllllllloooooooooooooooooooooooooooooooooooooooooooohhhhhhhhhhhhhhhhhhhhhhhhhoooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwaaaaaaaaaarrrrrrrrreyoubro.txt\" : {$message}"}
https://github.com/owncloud/core/blob/master/lib/private/Files/Storage/Local.php#L269 is where the problem caused. The reason is for rename/move operation the target file has ".d" + timestamp associated with the file name. This makes the filename long ( I am using a filename which has 249 char length for testing ).

sharidas on 11 Dec 2017

Hmmmm so this means we have other hidden issues with longer file names... Same will likely happen for versions.

PVince81 on 12 Dec 2017

@sharidas I suggest you ignore the trashbin/version issue for now as it would be for a different scope...

I have the feeling that we need to rework how we store any data that currently reuses the file name and makes it longer like encryption keys, trashed files, trashed versions, etc...

PVince81 on 12 Dec 2017

In theory for trashbin/versions, the data is anyway stored separately and not visible for end-users, so if we'd use md5's there as well it wouldn't hurt. I've raised https://github.com/owncloud/core/issues/29819 to look into this.

PVince81 on 12 Dec 2017

@sharidas by the way, we already had a PR https://github.com/owncloud/core/pull/26978. Please either reuse if that makes sense or close it.

Let's focus only on the upload case here for now.

PVince81 on 12 Dec 2017

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

ownclouders on 22 Jan 2018

@IljaN @PVince81 Can we keep this open?

cdamken on 22 Jan 2018

👍1

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

ownclouders on 2 Mar 2018

We will have much more trouble with md5 hashes, this is not really a solution IMHO.
The simple solution is, when there is an issue (in FS or DB) with handling too long filenames / pathes, use shorter filenames / pathes.

I don't see clean / other solutions for now.