Server: Can't download large folders (~>1GB) - Error "Argument 2 passed to OC\Streamer::__construct() must be of the type integer, float given..."

[alejandrocq]

Steps to reproduce

1. Try to download a large folder (>1GB).
2. Error is thrown while trying to generate a zip file.

Expected behaviour

Zip file should be generated correctly and downloaded correctly.

Actual behaviour

An error is thrown to the user with a reqId to request support.

Server configuration detail

Operating system: Linux 4.9.0-8-686-pae #1 SMP Debian 4.9.130-2 (2018-10-27) i686

Webserver: nginx/1.10.3 (fpm-fcgi)

Database: mysql 10.1.26

PHP version:

7.0.30-0+deb9u1
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, cgi-fcgi, mysqlnd, PDO, xml, apcu, apc, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, igbinary, imagick, imap, intl, json, exif, mcrypt, mysqli, pdo_mysql, pdo_sqlite, Phar, posix, readline, redis, shmop, SimpleXML, sockets, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Zend OPcache

Nextcloud version: 14.0.3 - 14.0.3.0

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: Nextcloud official web

Signing status

List of activated apps

Enabled:
 - accessibility: 1.0.1
 - activity: 2.7.0
 - calendar: 1.6.3
 - cloud_federation_api: 0.0.1
 - comments: 1.4.0
 - contacts: 2.1.6
 - dav: 1.6.0
 - federatedfilesharing: 1.4.0
 - federation: 1.4.0
 - files: 1.9.0
 - files_external: 1.5.0
 - files_pdfviewer: 1.3.2
 - files_sharing: 1.6.2
 - files_texteditor: 2.6.0
 - files_trashbin: 1.4.1
 - files_versions: 1.7.1
 - files_videoplayer: 1.3.0
 - firstrunwizard: 2.3.0
 - gallery: 18.1.0
 - issuetemplate: 0.4.0
 - logreader: 2.0.0
 - lookup_server_connector: 1.2.0
 - nextcloud_announcements: 1.3.0
 - notes: 2.4.2
 - notifications: 2.2.1
 - oauth2: 1.2.1
 - password_policy: 1.4.0
 - provisioning_api: 1.4.0
 - serverinfo: 1.4.0
 - sharebymail: 1.4.0
 - support: 1.0.0
 - survey_client: 1.2.0
 - systemtags: 1.4.0
 - tasks: 0.9.7
 - theming: 1.5.0
 - twofactor_backupcodes: 1.3.1
 - updatenotification: 1.4.1
 - workflowengine: 1.4.0
Disabled:
 - admin_audit
 - encryption
 - user_external
 - user_ldap

Configuration (config/config.php)

{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "***REMOVED SENSITIVE VALUE***"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
    "dbtype": "mysql",
    "version": "14.0.3.0",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "logtimezone": "UTC",
    "installed": true,
    "loglevel": 0,
    "memcache.local": "\\OC\\Memcache\\APCu",
    "maintenance": false,
    "theme": "",
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "PLAIN",
    "mail_smtpsecure": "tls",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpauth": 1,
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "587",
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "updater.secret": "***REMOVED SENSITIVE VALUE***"
}

Are you using external storage, if yes which one: No

Are you using encryption: No

Are you using an external user-backend, if yes which one: No

Client configuration

Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36

Operating system: Mac OS X Mojave 10.14.1

Logs

Browser log

Nextcloud log

TypeError: Argument 2 passed to OC\Streamer::__construct() must be of the type integer, float given, called in /var/www/html/owncloud/lib/private/legacy/files.php on line 166
/var/www/html/owncloud/lib/private/legacy/files.php - line 166:

OC\Streamer->__construct(OC\AppFramework\Http\Request {}, 12288299774, 893)

/var/www/html/owncloud/apps/files/ajax/download.php - line 64:

OC_Files::get("__SENSITIVE_INFO_HIDDEN__", "__SENSITIVE_INFO_HIDDEN", { head: false})

/var/www/html/owncloud/lib/private/Route/Route.php - line 155:

require_once("/var/www/ht ... p")


OC\Route\Route->OC\Route\{closure}("*** sensiti ... *")

/var/www/html/owncloud/lib/private/Route/Router.php - line 297:

call_user_func(Closure {}, { _route: "f ... "})

/var/www/html/owncloud/lib/base.php - line 987:

OC\Route\Router->match("/apps/files/ajax/download.php")

/var/www/html/owncloud/index.php - line 42:

OC::handleRequest()

Browser log

I have tried with several folders, but I always get the same problem. Notice that is a 32-bit server with latest Debian version.

[nextcloud-bot]

GitMate.io thinks possibly related issues are https://github.com/nextcloud/server/issues/8964 (Argument 2 passed to OC_App::isAppCompatible() must be of the type array, null given), https://github.com/nextcloud/server/issues/10601 (TypeError: Argument 8 passed to OCAFiles_SharingControllerShareAPIController::__construct() must be of th...), https://github.com/nextcloud/server/issues/3376 (Argument 3 passed to OCA\\Files\\Service\\TagService::__construct() must be an instance of OCP\\ITags), https://github.com/nextcloud/server/issues/2297 (Argument 1 passed to OCFilesConfigCachedMountInfo::__construct() must implement interface OCPIUser, null given), and https://github.com/nextcloud/server/issues/8489 (cannot download large folder with lots of small files).

[kesselb]

PHP_INT_MAX is 2147483647 for 32-bit. Looks like a value like 12288299774 is casted to float when greater than int max. I guess we could allow integer or float as value. Just wondering why you run a 32-bit linux on hardware with more than 4gb memory?

[kesselb]

https://github.com/nextcloud/server/blob/2e36069e24406455ad3f3998aa25e2a949d1402a/lib/private/Streamer.php#L46

Does it work when you change int $size to $size?

[alejandrocq]

Hi @danielkesselberg ! Thanks for your help. I have tried that, but for a folder with 4,2 GB size I get an incomplete .tar file. I can extract almost all files from it but then I get an error.

I'm using 32-bit with 2 GB of RAM, not 4 GB.

[wxiaoguang]

I think you meet the hard limit of the 32bit system (maybe not easy to resolve). If you are dealing with large files, it's better to have a 64bit system.

[sertraline]

Same here. It seems like I can download folders up to 2 GB, which is my RAM size, and cannot download anything larger than this. Mint 18.3 x32

[myevit]

Same here. It seems like I can download folders up to 2 GB, which is my RAM size, and cannot download anything larger than this. Mint 18.3 x32

same here - Physical Memory Usage 1.91 GB / 2 GB (95.59%) during downloading 5GB file then random drop on 500mb - 1.6gb

[StreamThreader]

Slackware 14.2 x86_64
Nextcloud 15.0.5
If download from local net (1Gbit link), all ok
If download through internet 50Mbit link, zip drop on 1GB (folder larger than 1.5GB).
HDD and RAM have free space.
Just files larger than 2GB downloaded without problem.
This problem with ZIP stream

[Elerir]

Operating System: Raspbian GNU/Linux 9 (stretch)
Kernel: Linux 4.14.98-v7+
Nextcloud 15.0.7
Doesn't seem linked to RAM.. my raspberry still has >60% free RAM when nextcloud displays the error and cancels the request

{"Exception":"TypeError","Message":"Argument 2 passed to OC\\Streamer::__construct() must be of the type integer, float given, called in \/nas\/data\/nextcloud\/lib\/private\/legacy\/files.php on line 166","Code":0,"Trace":[{"file":"\/nas\/data\/nextcloud\/lib\/private\/legacy\/files.php","line":166,"function":"__construct","class":"OC\\Streamer","type":"->","args":[{"__class__":"OC\\AppFramework\\Http\\Request"},13732257759,7612]},{"file":"\/nas\/data\/nextcloud\/apps\/files\/ajax\/download.php","line":64,"function":"get","class":"OC_Files","type":"::","args":["\/","Toutes photos",{"head":false}]},{"file":"\/nas\/data\/nextcloud\/lib\/private\/Route\/Route.php","line":155,"args":["\/nas\/data\/nextcloud\/apps\/files\/ajax\/download.php"],"function":"require_once"},{"function":"OC\\Route\\{closure}","class":"OC\\Route\\Route","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/nas\/data\/nextcloud\/lib\/private\/Route\/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"Closure"},{"_route":"files_ajax_download"}]},{"file":"\/nas\/data\/nextcloud\/lib\/base.php","line":987,"function":"match","class":"OC\\Route\\Router","type":"->","args":["\/apps\/files\/ajax\/download.php"]},{"file":"\/nas\/data\/nextcloud\/index.php","line":42,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"\/nas\/data\/nextcloud\/lib\/private\/Streamer.php","Line":46,"CustomMessage":"--"}

EDIT : I agree with wxiaoguang. That's a problem linked to 32bits systems. And since Raspbian is not available as 64bits we run a 32-bits apache/php instance...

[maranov]

Same with NC 16.

Operating System: Ubuntu 16.04, arm7l
Kernel: Linux 4.14.111-139
Nextcloud 16.0.3

{"Exception":"TypeError","Message":"Argument 2 passed to OC\\Streamer::__construct() must be of the type integer, float given, called in /var/www/nextcloud/lib/private/legacy/files.php on line 166","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/legacy/files.php","line":166,"function":"__construct","class":"OC\\Streamer","type":"->","args":[{"__class__":"OC\\AppFramework\\Http\\Request"},6974294953,3942]},{"file":"/var/www/nextcloud/apps/files/ajax/download.php","line":64,"function":"get","class":"OC_Files","type":"::","args":["/Photos","SPŠ",{"head":false}]},{"file":"/var/www/nextcloud/lib/private/Route/Route.php","line":155,"args":["/var/www/nextcloud/apps/files/ajax/download.php"],"function":"require_once"},{"function":"OC\\Route\\{closure}","class":"OC\\Route\\Route","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"Closure"},{"_route":"files_ajax_download"}]},{"file":"/var/www/nextcloud/lib/base.php","line":975,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/files/ajax/download.php"]},{"file":"/var/www/nextcloud/index.php","line":42,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/nextcloud/lib/private/Streamer.php","Line":46,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.102 Safari/537.36 Vivaldi/2.6.1566.44","version":"16.0.3.0","id":"5d3c19d192ad0"}

[scriptator]

I just updated from owncloud to the newest version of nextcloud (16.0.4) and experience the same issue. Is there anything I can do?

Raspbian GNU/Linux 9
Linux raspberrypi 4.19.66-v7+

TypeError: Argument 2 passed to OC\Streamer::__construct() must be of the type integer, float given, called in /var/www/nextcloud/lib/private/legacy/files.php on line 166

/var/www/nextcloud/lib/private/legacy/files.php - line 166:
OC\Streamer->__construct(OC\AppFramework\Http\Request {}, 3101495978, 9)

/var/www/nextcloud/apps/files/ajax/download.php - line 64:
OC_Files::get("/Fotos", "XXXX", { head: false})

/var/www/nextcloud/lib/private/Route/Route.php - line 155:
require_once("/var/www/ne ... p")

<<closure>>
OC\Route\Route->OC\Route\{closure}("*** sensiti ... *")

/var/www/nextcloud/lib/private/Route/Router.php - line 297:
call_user_func(Closure {}, { _route: "f ... "})

/var/www/nextcloud/lib/base.php - line 975:
OC\Route\Router->match("/apps/files/ajax/download.php")

/var/www/nextcloud/index.php - line 42:
OC::handleRequest()

[scriptator]

This commit fixes the problem for me: https://github.com/artonge/server/commit/435022515de1983f0fe3d3116acb71a0ed439693

[cquike]

Hi,
just for the record: I had the same issue also in Raspbian 32 bits with the download link in a shared folder even if it didn't exceed the 2GB limit. Applying the patch mentioned in the last comment also fixed the problem for me.

[Grokafar]

Same on my debian 32bits system the patch fixed the problem

[dd5fz]

Large directories not downloading in 18.0.6:

artonge@4350225 scheint auch bei 18.0.6 zu helfen.

[santa-klaus]

I have the same problem in 19.0.2 running on 32-bit RaspberryOS 10.4. It can't download a folder containing 380 photos (~3.2 GB).

If I apply https://github.com/artonge/server/commit/435022515de1983f0fe3d3116acb71a0ed439693 it works smoothly.

It would be great if this fix could be included in the next update.

[cClaude]

I confirm issue in 19.0.3 running on 32-bit Raspbian 10 (buster)

Look very similar to https://github.com/nextcloud/server/issues/15117

[nfp0]

I also have this problem on a 32-bit Armbian on an OdroidXU4.

[Wolfgang1966]

Same for me, Odroid HC1 (=XU4) with current Ubuntu. Using PHP 7.4 Fast CGI Module.

[Wolfgang1966]

The fix seems to be a one-liner basically:

In lib/private/Streamer.php

exchange
public function __construct(IRequest $request, int $size, int $numberOfFiles)

public function __construct(IRequest $request, float $size, int $numberOfFiles)

(line 56 in Nextcloud 19.0.3).

Didn't test that in depth, but at least got rid of the error message. The reason basically seems to be that PHP implicitely converts int to float as soon as a number overflow is about to occur. See https://www.php.net/manual/de/language.types.integer.php

[Wolfgang1966]

The fix seems to be a one-liner basically:

In lib/private/Streamer.php

exchange
public function __construct(IRequest $request, int $size, int $numberOfFiles)

public function __construct(IRequest $request, float $size, int $numberOfFiles)

(line 56 in Nextcloud 19.0.3).

Didn't test that in depth, but at least got rid of the error message. The reason basically seems to be that PHP implicitely converts int to float as soon as a number overflow is about to occur. See https://www.php.net/manual/de/language.types.integer.php

Have it now running this way fopr more than two weeks, no problems. Would recommend to use that fix.

Do I really have to open a PR for a one-liner change?

[nfp0]

@Wolfgang1966 This is awesome news! I hope it gets fixed ASAP.
Yes, please open a PR, as that will help the developers. Seems to be a simple fix, but they might not notice your posts.

[maranov]

@Wolfgang1966, not sure if floats can be safely used for size, their precision is limited. But I'm not a php dev.

Anyway, I did a bounty for this some time ago: https://www.bountysource.com/issues/66077825-can-t-download-large-folders-1gb-error-argument-2-passed-to-oc-streamer-__construct-must-be-of-the-type-integer-float-given, so whoever gets their PR merged...

[santa-klaus]

@Wolfgang1966 @nfp0
No, you don't have to open a PR, as there is one mentioned earlier in this issue. You can read more details about the current state and the problems with making $size a float there. I am not sure if there is anything planned for this issue at the moment though.
https://github.com/nextcloud/server/pull/16636

[nfp0]

@santa-klaus Oh, I'm sorry. I missed that.

[Wolfgang1966]

As far as I understand the code the number passed as argument is only used to decide what kind of archive to use, zip32 or zip64. So there is no real need to have a exact value. In principle one could just leave away the logic totally and always use zip64 as default, as far as there is no drawback known that may exist on client side.

[nfp0]

@Wolfgang1966 Can Zip64 be used on 32bit machines?

[Wolfgang1966]

Did a few more tests and found out that indeed Zip files above 4GB will be corrupted. Didn't notice that earlier because my previous tests were with folders above 2GB but below 4GB.

Changed the code now the following way:

                if ($size > 0 && $size < 4 * 1000 * 1000 * 1000 && $numberOfFiles < 65536) {
                        $this->streamerInstance = new ZipStreamer(['zip64' => false]);
//              } elseif ($request->isUserAgent($this->preferTarFor)) {
//                      $this->streamerInstance = new TarStreamer();
                } else {
                        $this->streamerInstance = new TarStreamer();
                }

Above 4GB the archive format is tar instead of zip that way. While there is no on board support for tar archives in Windows, there are may tools around providing support, and many users might have one installed already. If not, Windows 10 immediately offers to install one from the Microsoft store (at least it did that on my first attempt with a 9GB tar archive), so it should not be too difficult for the users to handle the archive. From my point of view this is an improvement compared to not being able to download it at all.