Server: Wrong file size shown and can't download files over ~4GB (64-bit server)

Created on 28 Oct 2017 · 20Comments · Source: nextcloud/server

The conclusion so far is that something prior to the oc_filecache table in the database is causing the issue, because the wrong value is in the table. A few other things that we checked looked right.

Steps to reproduce

Navigate to a file larger than ~4GB in nextcloud that's hosted on external storage
Observe that file size is wrong and looks like an overflowed 32 bit integer.

Expected behaviour

File size should be correct.

Actual behaviour

File size is not correct

Server configuration

Nextcloud version: 12.0.3.3
Operating system and version: Ubuntu 16.04 LTS
Apache version: 2.4.18
PHP version: PHP 7.0.22-0ubuntu0.16.04.1

Updated from an older Nextcloud/ownCloud or fresh install:
Fresh install of NC12 recently.

Where did you install Nextcloud from:
From the official channels.

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

"No errors have been found."

List of activated apps:
Enabled:

activity: 2.5.2
bruteforcesettings: 1.0.2
comments: 1.2.0
dav: 1.3.0
federatedfilesharing: 1.2.0
federation: 1.2.0
files: 1.7.2
files_downloadactivity: 1.1.1
files_external: 1.3.0
files_pdfviewer: 1.1.1
files_sharing: 1.4.0
files_texteditor: 2.4.1
files_trashbin: 1.2.0
files_versions: 1.5.0
files_videoplayer: 1.1.0
firstrunwizard: 2.1
gallery: 17.0.0
logreader: 2.0.0
lookup_server_connector: 1.0.0
nextcloud_announcements: 1.1
notifications: 2.0.0
oauth2: 1.0.5
password_policy: 1.2.2
provisioning_api: 1.2.0
serverinfo: 1.2.0
sharebymail: 1.2.0
survey_client: 1.0.0
systemtags: 1.2.0
theming: 1.3.0
twofactor_backupcodes: 1.1.1
twofactor_totp: 1.3.1
updatenotification: 1.2.0
workflowengine: 1.2.0
Disabled:
admin_audit
encryption
user_external
user_ldap

Nextcloud configuration:
{
"system": {
"instanceid": "ocxxg23ajtr3",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"10.0.200.10",
"snip"
],
"trusted_proxies": [
"10.0.200.1"
],
"forwarded_for_headers": [
"HTTP_X_FORWARDED_FOR"
],
"datadirectory": "\/nc-data",
"overwrite.cli.url": "https:\/\/10.0.200.10",
"htaccess.RewriteBase": "\/",
"dbtype": "mysql",
"version": "12.0.3.3",
"dbname": "nextcloud",
"dbhost": "localhost:3306",
"dbport": "",
"dbtableprefix": "oc_",
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"installed": true,
"mail_smtpmode": "smtp",
"mail_smtpauthtype": "LOGIN",
"mail_from_address": "cloud",
"mail_domain": snip,
"mail_smtphost": "10.0.200.13",
"mail_smtpport": "25"
}
}

Are you using external storage, if yes which one:
Yes, SMB

Are you using encryption:
No

Are you using an external user-backend, if yes which one:
No.

0. Needs triage bug external storage

Source

ShaunCurrier

All 20 comments

On further inspection, the file size problem is only happening in two folders in my external storage mount. These folders have 700-800 files in them each. When one of the problem files in these folders is moved to another location with fewer files in the same external storage mount, the size becomes correct in that new location.

Is there anything in the code that would be sensitive to the number of files in a directory?

ShaunCurrier on 31 Oct 2017

I've tried rescanning the files using occ files:scan and after a successful rescan, the files still showed incorrect size.

ShaunCurrier on 3 Nov 2017

I've verified that smbclient, when used manually, gives the correct size for the file.

ShaunCurrier on 3 Nov 2017

Can anyone point me where to look further into the code for where this problem might be? I've narrowed it to something between smbclient and wherever the size is inserted to the database. I've spent a couple hours trying to locate the issue in the code myself and, not being a php developer, I am having trouble making out the path between these two points. I'm taking any suggestions on how I could further investigate.

ShaunCurrier on 6 Dec 2017

Still broken on version 13.0.0 of Nextcloud Server.

I'm not a PHP developer, but I cobbled together a little php to test a theory that icewind's smb client might have something to do with this. It apparently did not, because the file sizes appear correctly when running the below code.

The following was used at an interactive prompt as reached by root@nextcloud:/var/www/nextcloud# sudo -u www-data php -a. PHP prompt removed for clarity.

include './apps/files_external/3rdparty/icewind/smb/src/Server.php';
use Icewind\SMB\Server;
use Icewind\SMB;
include './apps/files_external/3rdparty/icewind/smb/src/System.php';
require('apps/files_external/3rdparty/autoload.php');
$server = new Icewind\SMB\Server('myip', 'myusername', 'mypassword');
$share = $server->getShare('myshare');
$content = $share->dir('/mypath');
foreach ($content as $info) {
    echo $info->getName() . "\n";
    echo "\tsize :" . $info->getSize() . "\n";
}

The result:

...
myfile.ext
        size :22271073502
...

As you can see, the icewind library correctly detects the file as 22.2 GB file. The web gui shows this same file is 759.4 MB.

Can someone advise me where in the code I would best continue troubleshooting this? I now know that something goes wrong between this icewind library and the insert into the so-called "file cache". Where is that?

ShaunCurrier on 20 Feb 2018

👍1

Same Problem

pkkr on 11 Apr 2018

Still happening for me, too, on the latest (13.0.1).

Who can I communicate with that is familiar with external storage to point me in the right direction? I don't know PHP well, but I'm willing to try to fix it. I've already done some diagnostics and have book-ended where it could be.

ShaunCurrier on 12 Apr 2018

The obvious error is that the big file size from the icewind wrapper will be stored in a small integer somewhere and the information gets lost.

go2sh on 12 Apr 2018

Yes. This is happening somewhere between icewind and the database. The data type in the database is correct. I searched through all the different variables containing "size" and could not find where it was being improperly converted.

ShaunCurrier on 12 Apr 2018

👍1

@MorrisJobke I've seen you are involved in external storage. Can you provide your sage wisdom? As described above and in the community thread, I've narrowed this issue of incorrect SMB file sizes down. Somewhere after the icewind SMB library and prior to insertion in the database, the file size is getting slammed to a smaller integer. Can you think of any leads on where this could be happening?

ShaunCurrier on 6 Jun 2018

I want to be more precise about what is happening when attempting to download a file from the server when it is afflicted by this size issue. The Content-Length header in the GET response for the file shows the incorrect size. This causes the download to finish at the given byte length even though it is not the actual file size. So the downloaded file is missing data.

ShaunCurrier on 6 Jun 2018

cc @icewind1991 @rullzer

MorrisJobke on 6 Jun 2018

cc @danxuliu

MorrisJobke on 6 Jun 2018

Good news, I am learning me some PHP to continue on the journey of solving this. I now am set up with a trial of PHP Storm have it successfully connected to an xdebug installation on an image of my server.

I traced this to the Connection.read() method in the following file:
apps/files_external/3rdparty/icewind/smb/src/Connection.php

In that file, it seems that the smb machinery is responding to the allinfo command with the wrong size. I tested this theory with smbclient at the CLI and sure enough, the file size is coming out wrong from an allinfo command on a 22.2GB file:

smb: \> allinfo "mypath/myfile.ext"
altname: D6MTHJ~1.MKV
create_time:    Thu Jun  8 02:41:08 2017 UTC
access_time:    Thu Jun  8 03:30:37 2017 UTC
write_time:     Thu Jun  8 03:22:27 2017 UTC
change_time:    Thu Jun  8 03:22:27 2017 UTC
attributes: A (20)
stream: [::$DATA], 796237022 bytes

...but when I stat the file instead of allinfo it, I get the right size:

smb: \> stat "mypath/myfile.ext"
File: \mypath/myfile.ext
Size: 22271073502       Blocks: 43527489        regular file
Inode: 25187    Links: 1
Access: (0775/-rwxrwxr-x)       Uid: 1002       Gid: 1008
Access: 2017-06-08 03:30:37 +0000
Modify: 2017-06-08 03:22:27 +0000
Change: 2017-06-08 03:34:07 +0000

I did some Googling and it may have something to do with using ZFS on my SMB file server. (Additional reference: zfs_space)

That's where I'll have to end for tonight. To be continued...

ShaunCurrier on 13 Jun 2018

Aaah smb fun... let me summon @icewind1991

rullzer on 13 Jun 2018

I believe I just came across the same problem on a similar situation: 4.8 Gb file on external storage connected via SMB/CIFS. Download stops at the wrong reported size and file is therefore unusable. I haven't had the chance to deep dive like you have but let me know if you need me to run any logs for comparison.

Claudiohbsantos on 13 Jun 2018

@Claudiohbsantos, is your filesystem ZFS for the affected files?

On a separate note, I discovered that php-smbclient was not installed. The docs seem to prefer it. I installed it, restarted Apache, and rescanned. At first it seemed to not have fixed the issue, but now the next day it looks like it did fix it. I can see the full file size in limited testing. I'm sure the download will work now, too, but can't test since I'm on mobile. As a side benefit, performance seems to be much better as well.

But this doesn't fix the root cause of the original issue, which seems to lie somewhere from our interface to smbclient or beyond, perhaps all the way to the SMB server and its interactions with my ZFS filesystem.

So, @Claudiohbsantos, are you using ZFS?

ShaunCurrier on 13 Jun 2018

I've confirmed through further testing that installing php-smbclient, reloading apache, and rescanning the files allows >4GB files to work correctly. Note that this still doesn't fix the issue when the server is running smbclient.

ShaunCurrier on 14 Jun 2018

@ShaunCurrier : It is actually EXT4.

I'll give that a shot during the weekend when I'm not depending on the server to see if it works for me as well. Thanks for the tip.

Claudiohbsantos on 14 Jun 2018

I've confirmed through further testing that installing php-smbclient, reloading apache, and rescanning the files allows >4GB files to work correctly. Note that this still doesn't fix the issue when the server is running smbclient.

Solution, please use php-smbclient then :)