Client: Windows and Linux 32-bit clients crashing on large number of files

[montanaviking]

Expected behaviour

Tell us what should happen
Should sync any number of files up to physical memory/disk space of computers

Actual behaviour

Tell us what happens instead
Attempts to sync more than approximately 300,000 to 400,000 files fails with Windows and 32-bit Linux sync clients. The Windows sync client appears to crash when its RAM requirement exceeds approximately 1.5GB and while it's checking for things to sync - with an error from Visual C++. The 32-bit Ubuntu Linux client also crashes under similar conditions.
However, the 64-bit Linux sync client is handling > 900,000 files - about 50GB of data
Thanks

Steps to reproduce

1. Increase number of files to sync to >900,000
2. 
   
   
   
   1. 
      
      

      Server configuration

   
   

Operating system: Ubuntu 14.04 64-bit
3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Web server: Apache/2.4.7 (Ubuntu)

Database:mysql Ver 14.14 Distrib 5.5.52, for debian-linux-gnu (x86_64) using readline 6.3

PHP version: PHP 5.5.9-1ubuntu4.20 (cli) (built: Oct 3 2016 13:00:37)

ownCloud version: 9.1.1

Storage backend (external storage): None

Client configuration

Client version: 2.2.4 build 6408

Operating system: Windows 7

OS language: English

Installation path of client: C:\Program Files (x86)\ownCloud

Logs

Please use Gist (https://gist.github.com/) or a similar code paster for longer
logs.

Template for output < 10 lines

1. Client logfile: Output of owncloud --logwindow or owncloud --logfile log.txt
   (On Windows using cmd.exe, you might need to first cd into the ownCloud directory)
   (See also http://doc.owncloud.org/desktop/2.2/troubleshooting.html#client-logfile )
2. Web server error log:
3. Server logfile: ownCloud log (data/owncloud.log):

[guruz]

(Also posted to central on https://central.owncloud.org/t/owncloud-client-crashes-windows/3081/1 )

FYI @jturcotte

[michaelstingl]

As I understood this happens due to technical limitations of the 32-bit builds.

@guruz What are the plans for building 64-bit for Windows?

00006335

[montanaviking]

Hi Michael,
Would it be possible for me to compile the 64-bit client for Windows to overcome the limit? If so, are there build instructions?
Thanks,
Phil

[michaelstingl]

@montanaviking I'm not the expert for building the desktop client. Here you can find some information: https://doc.owncloud.org/desktop/2.2/building.html

[guruz]

(Edited the link above to be for 2.2 not 2.0)

[guruz]

Did 64-bit help on Linux?

[montanaviking]

Hi Markus,
The sync client worked perfectly on 64-bit Ubuntu 14.04 Linux i.e. version:
2.2.4 Built from Git revision eaeed0 on Oct 11 2016, 22:30:10 using Qt 4.8.6, OpenSSL 1.0.1f 6 Jan 2014
However, the 2.24 versions for 32-bin Ubuntu 14.04 Linux and Windows 7 crashed as of about 3wks ago when the filecounts exceeded about 400K files. I am running the 64-bit clients at > 1 million files without a problem.
Thanks again,
Phil

[scambra]

I have same issue, is there any plan to have win-64 version? Or instructions to compile a 64-bit client for windows

[guruz]

@scambra You could attempt a build with MSVC.
The non-MSVC instructions are at https://doc.owncloud.org/desktop/2.2/building.html#windows-development-build
Please keep us updated if you suceed

[ogoffart]

Since all at least all file names need to be in memory, and some more overhead for each files, this sets an upper limit on the amount of files that can be synced to fit in the memory available per process on 32 bit builds. We can try to reduce a bit the memory usage of the client but this is not something that will change in the next version.
So the only way to fix this is to do 64bit builds.

This is a packaging issue that is being tracked in https://github.com/owncloud/client/issues/2717

Other than that, should this issue be closed?

[guruz]

@ogoffart We had some discussions yesterday on IRC about this
64bit does not depend on doing a MSVC build, this is only one way. @dschmidt was thinking of still doing it the mingw cross-compile way.
But all ways would involve quite some work in Jenkins, docker, NSIS etc. So we're still thinking of memory usage reduction is easier.

FYI @dschmidt @jturcotte

[montanaviking]

Hi Olivier,
That makes sense. It probably also explains why sync clients from Google
crash on large numbers of files too.
I look forward to either being able to do a 64-bit Windows client build or
having one available for Windows. I use the 64bit client on my Linux
boxes, but others on our team use Windows so see the file limits.
Thanks,
Phil

[montanaviking]

I think the original issue can be closed, since the cause is known, but any
issues with building the 64-bit Windows clients still might need to be
resolved and left open.
Thanks so much,
Phil

On Fri, Jan 13, 2017 at 8:39 AM, phil marsh [e-mail] wrote:

Hi Olivier,
That makes sense. It probably also explains why sync clients from Google
crash on large numbers of files too.
I look forward to either being able to do a 64-bit Windows client build or
having one available for Windows. I use the 64bit client on my Linux
boxes, but others on our team use Windows so see the file limits.
Thanks,
Phil

On Fri, Jan 13, 2017 at 5:58 AM, Olivier Goffart <[email protected]

wrote:

Since all at least all file names need to be in memory, and some more
overhead for each files, this sets an upper limit on the amount of files
that can be synced to fit in the memory available per process on 32 bit
builds. We can try to reduce a bit the memory usage of the client but this
is not something that will change in the next version.
So the only way to fix this is to do 64bit builds.

This is a packaging issue that is being tracked in #2717
https://github.com/owncloud/client/issues/2717

Other than that, should this issue be closed?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/owncloud/client/issues/5237#issuecomment-272448581,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHpr2hWixUNYrBodhGnIAsNiq5MdX7WEks5rR4LzgaJpZM4KTFso
.

[guruz]

This will be vastly better already with 2.3 thanks to several memory usage improvements in there. :)

[inayet]

Hello everybody, the solution that worked for me was the clear ownCloud folder in the %appdata% destination

[michaelstingl]

Windows 64bit builds available here:
https://download.owncloud.com/desktop/ownCloud/testing/2.7.0.2141-v270beta5/win/