Client: Windows client 2.1.1 crashes with out of memory when commencing data transfer

[fowie]

Expected behaviour

Sync should finish successfully (and does on other machines)

Actual behaviour

On my windows 10 machine, own cloud starts up, performs a scan, then prepares to upload or download modified files. As soon as this begins, owncloud crashes with a Visual C++ Runtime error, stating "the application has requested the runtime to terminate it in an unusual way" (about 10 of these popups occur before owncloud finishes crashing). I have tried uninstalling and reinstalling the client. Uninstall, reboot, reinstall. Uninstalling all Visual C++ redistributables and owncloud, then reinstalling owncloud (letting it install the VCRedist), uninstalling the VC redist and installing my own fresh copy from Microsoft. All result in the same crash. Are there registry keys I can look for to make sure it's all removed completely, or is there a stack trace you'd like me to collect?

Steps to reproduce

1. Start owncloud
2. Wait for sync transfers to begin
3. 
   

   Server configuration

Operating system:
Ubuntu 14.04 LTS
Web server:
Apache 2.4.7
Database:
MySQL
PHP version:
PHP/5.5.9-1ubuntu4.14
ownCloud version:
9.0.0
Storage backend:
ZFS

Client configuration

Client version:
Windows client 2.1.1 (build 5837)
Operating system:
Windows 10 Pro N
OS language:
English
Installation path of client:
C:\Program Files (x86)\ownCloud

Logs

1. Output of owncloud --logwindow or owncloud --logfile log.txt
   Didn't see much useful in here, and the latest log is >100MB, so I've cropped to the last 4,000 ish lines, but: https://gist.github.com/fowie/2db5136999f0150a4772
2. Web server error log:
   [EMPTY] no errors from Apache
3. ownCloud log (data/owncloud.log):
   Same size log here, here's the last few thousand...
   https://gist.github.com/fowie/d6d07575e0df5d36b650

[Torashin]

I'm having the same issue syncing locally to a Bananan server from Win10.

[sirio3mil]

Same issue with same client version on Windows Server 2012 R2 and Windows Server 2008 R2. I have multiples crash, my sync folder is really big, not in size but in quantity, more than 1 million files and half million folders. Make the upload for first time take me more than 5 days cuz clients some times crash once upload 8 GB, or searching for changes.

[sirio3mil]

windows's log is not very useful

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2016-03-29T08:28:07.000000000Z" /> 
  <EventRecordID>1391</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>WIN-IUKESR0D9L2</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>owncloud.exe</Data> 
  <Data>2.1.1.5837</Data> 
  <Data>00000000</Data> 
  <Data>libstdc++-6.dll</Data> 
  <Data>0.0.0.0</Data> 
  <Data>35d135ab</Data> 
  <Data>40000015</Data> 
  <Data>00021d98</Data> 
  <Data>508</Data> 
  <Data>01d18944fad4f5e2</Data> 
  <Data>C:\Program Files (x86)\ownCloud\owncloud.exe</Data> 
  <Data>C:\Program Files (x86)\ownCloud\libstdc++-6.dll</Data> 
  <Data>298b6ccf-f588-11e5-80c5-005056b26f8e</Data> 
  <Data /> 
  <Data /> 
  </EventData>
  </Event>

[sirio3mil]

App crash window's error log:

Firma con problemas:
Nombre del evento de problema: APPCRASH
Nombre de la aplicación: owncloud.exe
Versión de la aplicación: 2.1.1.5837
Marca de tiempo de la aplicación: 00000000
Nombre del módulo con errores: libstdc++-6.dll
Versión del módulo con errores: 0.0.0.0
Marca de tiempo del módulo con errores: 35d135ab
Código de excepción: 40000015
Desplazamiento de excepción: 00021d98
Versión del sistema operativo: 6.3.9600.2.0.0.272.7
Id. de configuración regional: 3082
Información adicional 1: ded7
Información adicional 2: ded79d95e5579864469694df8c3859f9
Información adicional 3: b843
Información adicional 4: b84301101d0aaf37ea196b6ce986624b

Lea nuestra declaración de privacidad en línea:
http://go.microsoft.com/fwlink/?linkid=280262

Si la declaración de privacidad en línea no está disponible, lea la declaración de privacidad sin conexión:
C:\Windows\system32\es-ES\erofflps.txt

[fowie]

Looks like in my case the problem was one of two things.
1 - Some computer-challenged person tried to sync up a file with the oddest name I've ever seen, it included an opening ", a parenthesis, and a bunch of commas and periods (basically it was a quoted lyric from the song). I think that was causing one of the crashes
2 - The user's local hard disk they were pulling data from to go into the cloud was in the process of dying, causing uploads to not complete. I'll try to dig up more details and post here.

[sweeperoad]

I've the same problem.

When the Windows client (2.1.1 (build 5837)), running in a Windows 10 Home, is looking for changes in the folder, the client crashes with runtime error. The server is running in Debian, and the owncloud version is 8.

If anyone needs files to examine the error, just tell.

[ctastad]

I started experiencing this problem over the weekend while I was away.

Running:
owncloud 8.2.3
raspberry pi 3
MySQL DB

My setup is very new at approximately 1 week old, so I have only ever had one build of the sync client installed.

After experiencing the error I did a bit of shuffling and found that deleting a folder of torrented music files that was among what was being uploaded allowed the sync client to continue after it was restarted. I would assume some of these music files violated the naming convention mentioned by fowie. The client progressed fine for a while, but I have since hit the runtime error again and haven't been able to debug.

Also, I wasn't able to find anything of note in the owncloud.log file.

[Starlord006]

I've installed V.2.0.2 of the client, to get rid of this ERROR

So far the transfer restarted flawless

[sweeperoad]

Installed the version 2.0.2, but the client still crashes with runtime error.

[elm13]

I am experiencing the same error since approximately mid-February 2016.
Windows 7 Professional
ownCloud Client v2.1.1 (build 5837)

[dragotin]

have you tried to uninstall the client completely and do a fresh install? There was a known error about when the client was updated.

[elm13]

yes, I have uninstalled the client and reinstalled twice or three times already, did not help.

[sweeperoad]

I unistalled the client and installed three times too, and the problem ísn't solved.

[madorian]

I got client on Windows-10, 64 with owncloud-9 and I get the same thing. It starts sync'ing, then displays this message all over te screen, like 15 times, then whole computer doesn't respond at all and has to be rebooted.

[madorian]

I installed 2.0.2 and it seems to be working fine now, but it says "update 2.1.1.5837 is available. Restart to start update". I hope it doesn't update itself and run into it again. Any way to disable automatic update?

[stecklars]

I can confirm this issue, I've got the same problem. It happens on a large music folder with lots of files.

[dragotin]

Guys, please, we should find out why it crashes, rather than working around it with an ancient client.

Please try version 2.2.0, and see if it still crashes.

[stecklars]

Yes, this is happening on 2.2.0 for me, I just started using OwnCloud yesterday and downloaded the current client. I am using Windows 10 64 bit.

Event log:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2016-05-18T03:12:55.264843700Z" /> 
  <EventRecordID>3862</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>Lars-PC</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>owncloud.exe</Data> 
  <Data>2.2.0.6076</Data> 
  <Data>00000000</Data> 
  <Data>libstdc++-6.dll</Data> 
  <Data>0.0.0.0</Data> 
  <Data>35d135ab</Data> 
  <Data>40000015</Data> 
  <Data>0001941d</Data> 
  <Data>da8</Data> 
  <Data>01d1b0b2d73b3c3e</Data> 
  <Data>C:\Program Files (x86)\ownCloud\owncloud.exe</Data> 
  <Data>C:\Program Files (x86)\ownCloud\libstdc++-6.dll</Data> 
  <Data>d8572f5a-8f9e-4ad0-837e-dce2ae9fac7f</Data> 
  <Data /> 
  <Data /> 
  </EventData>
  </Event>

Also, this is what it looks like:
http://i.imgur.com/GNhT5ZA.jpg

[jturcotte]

@stecklars Were you able to see the crash reporter window? Or can you only see the windows runtime error window?

If you can click the "Send this report" button and tell us the time you did, we could fetch that report on our server to get a stack trace.

[stecklars]

@jturcotte There was no crash reporter window, so unfortunately I can't do that.

[sirio3mil]

Same for me, just updated to 2.2.0 and still crashing. No crash reporter window.

[jturcotte]

Looking at the log from @fowie and @sirio3mil 's https://github.com/owncloud/client/issues/4611#issuecomment-202427627 it seems like both have huge amounts of files and directories to sync, and it seems to crash more on initial syncs. Is this the case for everybody?

@mcastroSG Do you know if we have a setup available where we could test for a 1 million files sync?

[jturcotte]

Did anybody also notice huge memory usage? Since we ship 32bit binaries, the process will crash if the used memory crosses the 2GB limit.

[sirio3mil]

you are right @jturcotte I have huge amount of files and folders, and I detected that when app finish to sync it crash, so it search cross hundred folder, send changes and crash.

[stecklars]

@jturcotte Yes. For me it was an initial sync with 70 folders containing 1063 files making up for 7.67 GB.

I was able to see huge memory usage right before it crashed, can't exactly remember how much it was, but it was way above 1 GB. I might just try again today and report back later.

[stecklars]

@jturcotte I recorded this with perfmon. It starts shortly after starting the sync. You can clearly see that it suddenly uses a big amount of memory and then crashes:

[jturcotte]

@stecklars Wow, thanks. 1000 files isn't an extraordinary amount of files though, could you also share a logfile, like specified in the first comment/description of this issue to see if we can find any hint on what causes the issue, or to suggest that it might be cause by something else?

[TerryHewitt]

I too see a big jump in memory used in the last few seconds before it crashes. Something around 2GB jump shown in Task Manager
Best wishes
Terry

[jturcotte]

Here is a 2.2.1 nightly build that has a potential fix, if anybody having the issue with 2.2 could try with this build, it would allow us to know if the issue needs more work:

https://download.owncloud.com/desktop/daily/ownCloud-2.2.1.6102-nightly20160520-setup.exe

[sirio3mil]

Not working for me.

Initial assigned memory (MB) 200 aprox then go up to 1,3 GB, not respond but upload 74 MB in 2050 files, when finish assigned memory go down to 588 MB.

After some seconds search for changes again starting in 590 MB up to 1,3 GB, not respond but download 4 MB in 43 files, when finish assigned memory go down to 620 MB.

Some seconds after search for changes again starting in 620 MB aprox, but when finished (assigned memory go up to 1,45 GB) crash again, there is nothing to upload or download but crash.

I hope it help you.

- 
- 
   
  1000 
  2 
  100 
  0x80000000000000 
   
  2430 
  Application 
  WIN-IUKESR0D9L2 
   
  
- 
  owncloud.exe 
  2.2.1.6102 
  00000000 
  libstdc++-6.dll 
  0.0.0.0 
  35d135ab 
  40000015 
  00021d98 
  98c 
  01d1b2a82c4539d9 
  C:\Program Files (x86)\ownCloud\owncloud.exe 
  C:\Program Files (x86)\ownCloud\libstdc++-6.dll 
  5771cb16-1e9f-11e6-80c7-005056b26f8e 
   
   
  
  

Regards

[jturcotte]

@sirio3mil OK thanks for the input. Honestly I don't know if we ever supported 1M files syncs, so I would still like to know if other people could see improvements with the new build.

Maybe we should support that much files though (it's especially difficult on the first sync), but if you're hitting a different source of the crash, maybe we should track it in a different issue so that we can address the regression faster. Let's see if we get feedback from @stecklars, @fowie or others.

[TerryHewitt]

I had managed to isolate the problem to two folder trees. On 2.2.0 each failed with the above bug.
I've just downloaded the 2.2.1.nightly build, and tried it out and ....

One folder tree synced OK.

The other folder tree ...
keep getting a) "not responding"
and got a couple of b) "connected to" goes red and says not connected. Re-connects automatically.

The log page scrolls messages, then freezes for say 5-10 seconds, then carries on again.

But after about 30 minutes and 20GB copy, it's synced the second problematic folder tree. Hoorah.

Across 15 sync points (folder trees) its now syncs 833GB in 190,000 files in 26,000 folders
I'll leaving it running to see it it still works over the next few days.

Do you want me to keep using this version?
I'm happy to try subsequent builds if needed.

Many thanks for a prompt fix, much appreciated.

Best wishes
Terry

[TerryHewitt]

Jocelyn

It looks like it’s working. I’ll leave it running. See also my comments on GitHub directly.

Thanks for the prompt fix. Much appreciated.

Best wishes

Terry

Professor W T Hewitt

WTH Associates Ltd

Telephone: [phone]

Company registration number:7336226

HMRC VAT Registration number: 998 8457 23

Company Registered Address

St George's House | 215 - 219 Chester Road | Manchester | M15 4JE

From: Jocelyn Turcotte [mailto:[email protected]]
Sent: 20 May 2016 15:51
To: owncloud/client [email protected]
Cc: TerryHewitt [email protected]; Comment [email protected]
Subject: Re: [owncloud/client] Windows client 2.1.1 crashes with out of memory when commencing data transfer (#4611)

Here is a 2.2.1 nightly build that has a potential fix, if anybody having the issue with 2.2 could try with this build, it would allow us to know if the issue needs more work:

https://download.owncloud.com/desktop/daily/ownCloud-2.2.1.6102-nightly20160520-setup.exe

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub https://github.com/owncloud/client/issues/4611#issuecomment-220627156 https://github.com/notifications/beacon/AO-x87oPfhk36NeSXHDWSMMFQFwB3noWks5qDco4gaJpZM4H4676.gif

[Torashin]

The nightly build is working so far for me; thanks jturcotte!

[stecklars]

@jturcotte Yeah, it is fixed now. The client is using A LOT less memory and disk i/o usage is much lower, too (that was extremely high as well in the older version)

Thank you!

[jturcotte]

Thanks to everybody for verifying, you'll have to thank @ogoffart for saving the day on this one. I'm going to close this issue as fixed.

@sirio3mil Your issue is probably a different one then. The most files I've heard somebody trying to sync in one shot was 600k files in #2902 a while ago. It would be nice if you could try to add files progressively to your synced folder, like in chunk of 100k or 200k files, finish the sync, and then add new files in order to reduce the amount of files that the sync client to manage in memory all at once.

It's something that ideally shouldn't crash, but we'd need to make compromises on the conflict resolution reliability, or to do some major changes in the sync engine to support that many synced files at once, so I'm not sure if we can fix this short term.

[sirio3mil]

Thanks @jturcotte, just to clarify, surely I have badly explained above, the million and a half files are already synchronized, the problem with owncloud`s client is when adding new files. But ok I understand that do not give support to such a large volume of files although it would be an ideal functionality especially in my case that there are no deletions or modifications, is just add.

Regards

[deanpcmad]

That nightly build also fixes this issue for me, thanks!

[RedKage]

Hello there just had this issue a couple days ago.
This occurs when ownCloud is trying to sync my Atom user folder (where there are 6,350 Files, 1,538 Folders).
Haven't tried the nightly yet, I will right after I post stuff here.

Computer is Win8.1 x64 with 16Gb ram.
client: ownCloud is 2.2.0 (build 6076) rev 49a678
server: ownCloud 9.0.2 (stable) on my gentoo box. Encryption is ON and Memcached is used.

Here are two screenshots of the m$ dialog error:

Here is the event viewer related error:

Faulting application name: owncloud.exe, version: 2.2.0.6076, time stamp: 0x00000000
Faulting module name: libstdc++-6.dll, version: 0.0.0.0, time stamp: 0x35d135ab
Exception code: 0x40000015
Fault offset: 0x00021d32
Faulting process id: 0xdb4
Faulting application start time: 0x01d1b606d43dba48
Faulting application path: C:\Program Files (x86)\ownCloud\owncloud.exe
Faulting module path: C:\Program Files (x86)\ownCloud\libstdc++-6.dll
Report Id: 5759d4ce-21fb-11e6-84f6-5cf370615742
Faulting package full name: 
Faulting package-relative application ID: 

Here is another part of the event viewer log error:

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: owncloud.exe
P2: 2.2.0.6076
P3: 00000000
P4: libstdc++-6.dll
P5: 0.0.0.0
P6: 35d135ab
P7: 40000015
P8: 00021d32
P9: 
P10: 

Attached files:

These files may be available here:
C:\Users\RedShadow\AppData\Local\Microsoft\Windows\WER\ReportQueue\AppCrash_owncloud.exe_e3df5211c732b2a2a2731b74f6af95e43763f76_43d8d52d_268c6a45

Analysis symbol: 
Rechecking for solution: 0
Report Id: 5759d4ce-21fb-11e6-84f6-5cf370615742
Report Status: 2097253
Hashed bucket: 

Here is the mentioned attached fille file (Report.wer):
Report.zip

And then I tried debugging with my VS2015, and here is the output window content:
vsdebug.txt

I will now try the nighlty posted above and report back.

EDIT: can't try the nighlty fix cause it's apparently gone: 404
I will try with ownCloud-2.2.1.6112-nightly20160525-setup.exe

[RedKage]

3 hours without a single crash. The client synchronizes correctly and all is fine.
ownCloud-2.2.1.6112-nightly20160525-setup.exe does fix this issue!
:)

[cyclops1982]

I've had the same issue on a windows 10 HP laptop from my sister. Started a new sync with 35GB of images to upload. It's about 18K files, so that's quite a lot.

The nightly build 20160526 (6114) seems to be stable for me.

I myself still run win 8.1 on a daily basis and sync all my work files with it. I'm so happy with this solution as it just works so well normally. Keep it up, really great guys!

[SurajVerma]

I am facing exactly the same issue, the link for the nightly build is not working at the moment, can you provide with a new link to download the build.

Thanks a ton for your help.

[cyclops1982]

@SurajVerma I used this link: https://download.owncloud.com/desktop/daily/
Then i ordered based on Last modified and then select the file starting with owncloud (not the testpilotcloud).

For me, that currently ends up in this link:https://download.owncloud.com/desktop/daily/ownCloud-2.2.1.6114-nightly20160526-setup.exe

[SurajVerma]

@cyclops1982 Thank you very much for your help, I got the link after some google before you posted and downloaded the testpilotcloud one, now I am downloading the nightly. :)

[NikolasMeyer]

Tried 2.2.1.6127
Everything works fine

[hutber]

I was also getting the same errors, across the screen multiple times. I'll try the upgrade and cross my fingers 💃

[sweeperoad]

I just downloaded the version 2.2.1 (build 6146), with Windows 8.1 Home, formatted in the weekend, but the problem persists. The application crashes before start the download.

The data I'm trying to sync has 35k folders, 3M files and 230gb.

[guruz]

@sweeperoad The reported crash here was for uploads not downloads. Can you please report a new issue?

[sweeperoad]

@guruz I wanna write upload. The issue is the same. My fault.

[jturcotte]

3M files is too many files metadata to keep in the client's memory at the same time. Could you try adding them to your sync folders in chunks of 100k to 500k files with a successful sync in-between?

[sweeperoad]

I'll try.

[sweeperoad]

@jturcotte I've compressed some folder, and my sync folder now "only" have 81Gb and 82k files.

Everything uploaded like a charm.

[scriptPilot]

I have the same problem with only 25.000 files ... a folder with images.

[dragotin]

@scriptPilot you need to specify your problem more detailed, as the one above seemed ready to be closed.

[guruz]

@scriptPilot Please create a new issue with detailed information about the issue.
Also make sure to be on 2.2.1

[guruz]

2.3.0 beta1 has HUGE memory improvements when syncing a lot of files.
https://owncloud.org/changelog/desktop/