Server: NC16.0.3 Upload fails with files 2GiB in 32bit instances (again)

[nachoparker]

NextCloudPi 1.15.0, NC16.0.3, Odroid HC2. Tested by uploading a 5GiB file through the web interface.

# uname -a
Linux nextcloudpi 4.14.127-odroidxu4 #61 SMP PREEMPT Mon Jul 1 20:41:09 CEST 2019 armv7l GNU/Linux

I know this was fixed by using chunking in NC13 but it is back now.

I can see the file being fully uploaded and then reconstructed in the destination folder (increasing size until full size).

Then the interface fails with this error and the file gets removed

NCP ticket

https://github.com/nextcloud/nextcloudpi/issues/948

[F1orian]

I get this error message when uploading through the web interface:
Odroid HC2 NC 16.0.3

And this when uploading through WebDAV:

[no app in context] Error: Sabre\DAV\Exception\BadRequest: expected filesize 2147483647 got 1861091328 at <>

1. /var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php line 156
   put(null)
2. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 1096
   createFile("Snap.7z", null)
3. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 525
   createFile("Snap.7z", null, null)
4. <>
   httpPut(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
5. /var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
   call_user_func_array([Sabre\DAV\CorePlugin {},"httpPut"], [Sabre\HTTP\Requ ... }])
6. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 479
   emit("method:PUT", [Sabre\HTTP\Requ ... }])
7. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 254
   invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
8. /var/www/nextcloud/apps/dav/appinfo/v1/webdav.php line 80
   exec()
9. /var/www/nextcloud/remote.php line 163
   require_once("/var/www/nextcl ... p")

PUT /remote.php/webdav/Snap.7z
from 192.168.178.104 by ncp at 2019-07-17T10:41:35+00:00

I just did a new clean installation and tried the web Interface upload again:

The second error appeared approximately 100 times in the log.

[zedocrob]

Same issue here on Odroid HC2:
1- i can drag'ndrop a 4Gb file in my Browser and download it back correctly, but always got this error in logs:
`[PHP] Error: file_put_contents(): content truncated from 3915768813 to 2147483647 bytes at /var/www/nextcloud/lib/private/Files/Storage/Local.php#488

MOVE /remote.php/dav/uploads/xxx/web-file-upload-31c5297bca63498f17fedc95638e6c6f-1563389990100/.file
from 192.168.0.254 by xxx at 2019-07-17T19:05:36+00:00``

2-Using a webdav client (cyberduck, transmit...) i can't upload it, file chucking bug...
3-Also, I can't delete the 4Gb file in trash, i have this error:
[remote] Error: TypeError: Return value of OCA\Files_Trashbin\Sabre\AbstractTrash::getSize() must be of the type int, float returned at <<closure>>

i have to force trashbin:cleanup to delete it

[violoncelloCH]

cc @nextcloud/server-triage

[kesselb]

@nachoparker @F1orian https://github.com/nextcloud/server/issues/15095 https://github.com/nextcloud/server/issues/16138

@zedocrob https://github.com/nextcloud/server/issues/13160

[F1orian]

@kesselb May I ask why this issue was closed? I didn't see a strong resemblance in the issues you linked.
Thanks anyways for looking into this!

[kesselb]

expected filesize 2147483647 got 1861091328

Same as #15095. You tried to disable mod_reqtimeout and http2?

[F1orian]

@kesselb I disabled both mod_reqtimeout and http2. Still getting an error message and the upload fails

[webdav] Fatal: Sabre\DAV\Exception\BadRequest: expected filesize 2147483647 got 1025335296 at <>

1. /var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php line 156
   put(null)
2. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 1096
   createFile("Lightroom.7z", null)
3. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 525
   createFile("Lightroom.7z", null, null)
4. <>
   httpPut(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
5. /var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
   call_user_func_array([Sabre\DAV\CorePlugin {},"httpPut"], [Sabre\HTTP\Requ ... }])
6. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 479
   emit("method:PUT", [Sabre\HTTP\Requ ... }])
7. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 254
   invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
8. /var/www/nextcloud/apps/dav/appinfo/v1/webdav.php line 80
   exec()
9. /var/www/nextcloud/remote.php line 163
   require_once("/var/www/nextcl ... p")

PUT /remote.php/webdav/Lightroom.7z
from 192.168.178.104 by ncp at 2019-07-25T09:24:10+00:00

[kesselb]

Confirmed for 17: https://github.com/nextcloud/server/issues/17485

[delfer]

Partial solution (file uploaded, but was truncated to 2GiB):

--- lib/private/Files/Storage/Local.php.orig    2019-10-09 15:40:47.571461212 +0000
+++ lib/private/Files/Storage/Local.php 2019-10-09 15:50:39.208620584 +0000
@@ -485,6 +485,12 @@
    }

    public function writeStream(string $path, $stream, int $size = null): int {
-       return (int)file_put_contents($this->getSourcePath($path), $stream);
+       $fz = fopen($this->getSourcePath($path), 'wb');
+       $count = 0;
+       while(($l=fread($stream, 65536))) {
+               $count += fwrite($fz, $l);
+       }
+       fclose($fz);
+       return (int)$count;
    }
 }

[kesselb]

Thanks for digging @delfer :+1:

You could try to https://github.com/nextcloud/server/blob/93de63777e4a519484ad8f50f7a8ee809697c20a/lib/public/Files/Storage/IWriteStreamStorage.php#L39 remove : int there, in your above patch and return $count without the typecast.

[delfer]

@kesselb thank you for help. File is still truncated to 2GiB

-- lib/public/Files/Storage/IWriteStreamStorage.php.orig    2019-10-09 17:56:19.065499527 +0000
+++ lib/public/Files/Storage/IWriteStreamStorage.php    2019-10-09 17:56:36.965524710 +0000
@@ -36,5 +36,5 @@
     * @return int the number of bytes written
     * @since 15.0.0
     */
-   public function writeStream(string $path, $stream, int $size = null): int;
+   public function writeStream(string $path, $stream, int $size = null);
 }

--- lib/private/Files/Storage/Common.php.orig   2019-10-09 18:00:34.313848529 +0000
+++ lib/private/Files/Storage/Common.php    2019-10-09 17:58:42.645698305 +0000
@@ -820,7 +820,7 @@
     * @param int $size
     * @return int
     */
-   public function writeStream(string $path, $stream, int $size = null): int {
+   public function writeStream(string $path, $stream, int $size = null) {
        $target = $this->fopen($path, 'w');
        if (!$target) {
            return 0;

--- lib/private/Files/Storage/Local.php.orig    2019-10-09 15:40:47.571461212 +0000
+++ lib/private/Files/Storage/Local.php 2019-10-09 17:55:53.401463198 +0000
@@ -484,7 +484,13 @@
        }
    }

-   public function writeStream(string $path, $stream, int $size = null): int {
-       return (int)file_put_contents($this->getSourcePath($path), $stream);
+   public function writeStream(string $path, $stream, int $size = null) {
+       $fz = fopen($this->getSourcePath($path), 'wb');
+       $count = 0;
+       while(($l=fread($stream, 65536))) {
+               $count += fwrite($fz, $l);
+       }
+       fclose($fz);
+       return $count;
    }
 }

[delfer]

It can be fixed only by changes in PHP image. https://github.com/docker-library/php/pull/901

[delfer]

Fixed by docker-library/php#901
Tested in Docker image nextcloud:17-apache (nextcloud@sha256:58836d449406b396c1c119c2661577fd7d8284ff41a7d181a0e2be232d2757fc, 2019-10-24T05:04:49.946856094Z)
Can be closed.

[kesselb]

Can be closed.

nextcloud/docker :heavy_check_mark:
nextcoud/nextcloudpi :x: (does not inherit from library/php)

Probably they need to recompile php with the flags -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64. @nachoparker is it okay with you to move this issue to nextcloudpi?

Thanks @delfer for the research :+1:

[nachoparker]

already there https://github.com/nextcloud/nextcloudpi/issues/948

[kesselb]

Okay with you to close this issue then? @nachoparker @F1orian

[nachoparker]

I am not sure, rebuilding PHP is not always feasible, so if NC doesn't do anything about it the bug is going to remain there.

Ideally this would be worked around as it has been in the past.

[kesselb]

Index: lib/private/Files/Storage/Local.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- lib/private/Files/Storage/Local.php (revision 83af640780fd1532316b07bdfc68fd570c487a2c)
+++ lib/private/Files/Storage/Local.php (date 1572199369066)
@@ -218,7 +218,15 @@
    }

    public function file_put_contents($path, $data) {
-       return file_put_contents($this->getSourcePath($path), $data);
+       $handle = $this->fopen($path, 'w');
+       if ($handle === false) {
+           return false;
+       }
+
+       $result = fwrite($handle, $data);
+       fclose($handle);
+
+       return $result !== false;
    }

    public function unlink($path) {
Index: apps/dav/lib/Connector/Sabre/File.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- apps/dav/lib/Connector/Sabre/File.php   (revision 83af640780fd1532316b07bdfc68fd570c487a2c)
+++ apps/dav/lib/Connector/Sabre/File.php   (date 1572197457791)
@@ -172,7 +172,7 @@
                $this->changeLock(ILockingProvider::LOCK_EXCLUSIVE);
            }

-           if ($partStorage->instanceOfStorage(Storage\IWriteStreamStorage::class)) {
+           if (PHP_INT_SIZE === 8 && $partStorage->instanceOfStorage(Storage\IWriteStreamStorage::class)) {

                if (!is_resource($data)) {
                    $tmpData = fopen('php://temp', 'r+');

fopen and fwrite is able to write files bigger than PHP_INT_MAX (update: if php is compiled with -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64) . Additionally we need to disable IWriteStreamStorage for 32bit because the methods are type hinted. Don't know if such a patch is acceptable.

We probably could write some own storage adapter Local32 without the IWriteStreamStorage and all the knowledge how to read / write / determine file size / etc. for 32bit systems. Local would also benefit because some 32bit specific code could be removed.

Perhaps https://wiki.php.net/rfc/bigint will make it's way to php sometime.

[delfer]

@kesselb fopen and fwrite is NOT able to write files bigger than MAX_INT.
It is limitation of any program build with gcc, not only php.
You should look at my test algorithm and results in https://github.com/docker-library/php/pull/901

[kesselb]

Tested the above code on my rpi3 (with nextcloudpi) and were able to write files > 2147483647. Have you tried the above patch? Is there a way to figure out if php is compiled with -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64?

[delfer]

This code:

<?php
$f = fopen('largefile.tmp', 'wb');
for ($i = 0; $i < 2048+2; $i++) {
    $c = fwrite($f, pack('x1048576'));
    if ($c == 0) {
        trigger_error("Can not write large file", E_USER_ERROR);
    }
}
fclose ($f);

Command pack('x1048576') generates 1M of zeroes.

On Pi3 PHP compiled without flags after writing 2147483647 bytes writes 0 bytes every call. When compiled with flags - it writes as much bytes as in argument.

[kesselb]

Above code writes a 2149580800 bytes file. Added echo $i . PHP_EOL and it iterates till the end.

<?php
$f = fopen('largefile.tmp', 'wb');
for ($i = 0; $i < 4096+2; $i++) {
        echo $i . PHP_EOL;
    $c = fwrite($f, pack('x1048576'));
    if ($c == 0) {
        echo "zer0 bytes written" . PHP_EOL;
    }
}
fclose ($f);

Changed your code like above and run it again: Writes a 4297064448 bytes file.

pi@nextcloudpi:~ $ php -r "var_dump(PHP_INT_MAX);"
int(2147483647)
pi@nextcloudpi:~ $ php -r "var_dump(PHP_INT_SIZE);"
int(4)

[delfer]

@kesselb can you please check php -r "phpinfo();" | grep PHP_CFLAGS

[kesselb]

php -r "phpinfo();" | grep PHP_CFLAGS

PHP_CFLAGS is not present at phpinfo. It's the php version shipped by raspian buster.

[delfer]

@kesselb ok, try:

sudo apt-get install php-dev
php-config  --configure-options

I also checked docker run -ti --rm php sh -c 'php -r "phpinfo();" | grep FLAGS'

[kesselb]

php-config  --configure-options
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

Hmm. I'm not able to write files bigger than > PHP_INT_MAX without my above patch. I see the same "content truncated from" message.

@nachoparker @F1orian could one of you check if the php version installed on odroid hc2 is also compiled with -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64?

How do we proceed?

1) The basic requirement is a php version compiled with -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

2) Even with the above compiler flags a value > 2147483647 will always casted to float. We are able to use fopen and frwrite but not file_put_contents. It's not possible to use IWriteStreamStorage (type hints) but Local.file_put_contents is not type hinted.

Some other issues / pull requests related to 32bit.

https://github.com/nextcloud/server/pull/16636
https://github.com/nextcloud/server/pull/17609

Any ideas?

[delfer]

It is very strange because just after my PR https://github.com/docker-library/php/pull/901 was merged, I can upload and download files over 2G on my Odroid HC2 with latest Nextcloud docker image.

[kesselb]

Some docker related magic? :man_shrugging: I'm not using nextcloud via docker with nextcloudpi.

[ChristophWurst]

Does that mean we don't need https://github.com/nextcloud/server/pull/17609?

[kesselb]

Does that mean we don't need #17609?

No. Newer docker images (that are build from library/php) are able to write files bigger > 2gb with file_put_contents (log message: "content truncated from x to 2147483647" where x is > 2147483647).

17609 is a different story. 32bit php is still not able to have ints with values > 2147483647 so the typehint will break.

[thrubovc]

Yesterday I opened bug #18720 which was referenced to this one. What's funny is that I can upload/download anything below 4GB, not just 2GB as this bug reports. (dockerized Nextcloud on RPI4). Or am I just missing something?

[StableNarwhal]

Yesterday I opened bug #18720 which was referenced to this one. What's funny is that I can upload/download anything below 4GB, not just 2GB as this bug reports. (dockerized Nextcloud on RPI4). Or am I just missing something?

Did you fix it? Having the same problem

[thrubovc]

Nope, never got it to work on 32bit raspbian. I have, however, switched to Manjaro which is fully 64bit. Nextcloud is running on it flawlessly

[StableNarwhal]

Thank your for your answer. Guess I'll tinker around a bit then. Cheers!

[encodeering]

Hi,

also run into the trunction issue on a armhf odroid hc1 and just wanted to share my configuration, as it might help to solve this.

Versions

• PHP 7.3.18 (cli) (built: May 16 2020 14:51:26) ( NTS )
• Nextcloud 18.0.4

php -r "phpinfo();" | grep PHP_CFLAGS
PHP_CFLAGS => -fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
$_SERVER['PHP_CFLAGS'] => -fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

php -r "var_dump(PHP_INT_MAX);"
int(2147483647)

php -r "var_dump(PHP_INT_SIZE);"
int(4)

uname -m -r
4.14.176-1-ARCH armv7l

Upload of a ~3.5GB is fine at browser level, but the following erros occurs later

> file_put_contents(): content truncated from 3706877836 to 2147483647 bytes at /var/www/html/lib/private/Files/Storage/Local.php#503

Nextcloud is behind a nginx proxy and uses a php-fpm configuration.

Anything else that could be useful?

[vnorbi21]

hi guys, i have the same problem and i can't resolve it :(

Hardware: Raspberry pi4 2gb
Software: 32bit raspbian
Nextcloud 19.0.1 running on apache with php7.3

I already changed the php.ini settings, request timeout in apache, etc.. I saw that it was resolved in nextcloud 13, so it may be possible to fix it again? A cloud solution with a 2gb upload limit would be strange... :(

[thrubovc]

hi guys, i have the same problem and i can't resolve it :(

Hardware: Raspberry pi4 2gb
Software: 32bit raspbian
Nextcloud 19.0.1 running on apache with php7.3

I already changed the php.ini settings, request timeout in apache, etc.. I saw that it was resolved in nextcloud 13, so it may be possible to fix it again? A cloud solution with a 2gb upload limit would be strange... :(

get ubuntu. it's fully 64bit. it's a little heavier, but very similar to raspbian.

[vnorbi21]

I saw your comment about Manjaro, and I was thinking about changing OS for 64 bit. But I still have to confirm, that everything I use works well on ubuntu too (pihole, smtp, etc..)

BTW I'm still hoping to figure it out for Raspbian, as I did put a lot of energy in my current solution 😄 Speaking of Raspbian, Rpi OS 64 bit is still in a very early stage, I would not risk it for a daily use yet.

I did a little search, but didn't find the issue where uploading more than 2gb was solved in NC13 for 32 bit devices. I hope to find something in that version to get our problem solved again.

[vnorbi21]

I spent the last 2 days fixing this problem. The result was:

• There is no way to get nextcloud 19 working with >2gb files on Raspbian (or any 32bit arm based os basically).
• I tried to install an ubuntu arm64 version, but it is too much for the Raspberry pi4. After a few freezes, a lot of compatibility issues (no working vnc solution, lot of missing arm packages) i gave up, ubuntu is not the way to go.
• I came back to Raspbian 32 bit, and tried to downgrade my php version from 7.3 to 7.2. This caused a loop on the login screen of nextcloud.
• I tried php 7.3 with my owncloud instance. The login was fine, but file uploading behaved the same as in nextcloud (was not able to upload files bigger than 2gb).
• I downgraded php to version 7.2 on my owncloud instance. Suddenly, the problem was solved, i had no issues uploading my 2.5 gb test file.

Regarding the original nextcloud problem, the solution will be around php changes between version 7.2 and 7.3. I'll not dig deeper into this now, i'll stick with my owncloud for a while. I'll be happy to come back to nextcloud, if the problem will be solved :)

[thrubovc]

That's a bummer to hear. Ubuntu did feel a little heavier for rpi4, slower logins, services running by default that I didn't need, and the darn disks would never spin down... but I didn't have any major issues on it. I reverted to manjaro and its fast and I think the community is really good, including the upstream arch linux one.

[skjnldsv]

Hey! What is the status here?

[vnorbi21]

Hi,
it's all the same. the uploader doesn't like big files on any 32 bit operating system. :(
Tried it with Manjaro 32 bit and Raspbian 32 bit on a Raspberry pi4.

[thrubovc]

I wouldn't count on the Nextcloud team to fix this as 32bit OSes are slowly dying out and there are higher priorities than working on 32bit-related bugs. Thankfully you have a handful of fully 64bit OSes to replace Raspbian with (I presume you're running Raspbian).

[skjnldsv]

Yeah, unfortunately, I think I'll close this as wontfix. :/
Even arm are now almost all 64bits :shrug:

[pjft]

Even though this was closed recently, I just wanted to say that the code shared by @kesselb solves the problem perfectly on #22936 . Thank you.

[githule]

I wouldn't count on the Nextcloud team to fix this as 32bit OSes are slowly dying out and there are higher priorities than working on 32bit-related bugs. Thankfully you have a handful of fully 64bit OSes to replace Raspbian with (I presume you're running Raspbian).

Should tell this to lots of user that build/bought Pi based offers around nextcloud. Maybe we should update requierments or catch the exception...

[thrubovc]

I wouldn't count on the Nextcloud team to fix this as 32bit OSes are slowly dying out and there are higher priorities than working on 32bit-related bugs. Thankfully you have a handful of fully 64bit OSes to replace Raspbian with (I presume you're running Raspbian).

Should tell this to lots of user that build/bought Pi based offers around nextcloud. Maybe we should update requierments or catch the exception...

I am running a fully 64bit OS on my RPI. Nobody said you're stuck with 32bit if you bought a RPI.

[githule]

I wouldn't count on the Nextcloud team to fix this as 32bit OSes are slowly dying out and there are higher priorities than working on 32bit-related bugs. Thankfully you have a handful of fully 64bit OSes to replace Raspbian with (I presume you're running Raspbian).

Should tell this to lots of user that build/bought Pi based offers around nextcloud. Maybe we should update requierments or catch the exception...

I am running a fully 64bit OS on my RPI. Nobody said you're stuck with 32bit if you bought a RPI.

Ok if you have a recent pi. There are also alternate boards.
Maybe at least the UI must be aware of a max upload size / download setting ? Indeed, however a recent Pi or alternate board can technically handle big files, Its not a good thing to let people upload very big file considering the storage/computing capacity.