Ledger-live-desktop: absurd memory usage

[bumbar1]

Ledger Live Version and Operating System

• tested on Ledger Live 1.2.0
• Platform and version: elementary os 0.4.1

Expected behavior

app should play nicely with system, not abuse it

Actual behavior

app doesn't seem to stop consuming memory

Steps to reproduce the behavior

picture 1, after left running for a while

picture 2, restarted and said it was synchronized

picture 3, restarted after sync finished, kept resyncing, memory usage just goes up... (got a few http 500 errors on this stage)

[gre]

cc @KhalilBellakrid we need to investigate this. i suspect the sync is going wild

[juan-cortes]

@bumbar1 it would be interesting to know which cryptos you have installed when this happened. Syncing can trigger several processes depending on which account is being synced, perhaps there's a bug in a specific one and we can narrow it down faster.

[KhalilBellakrid]

@gre this is huge, you have to be sure that you kill spawned processes where libcore got instanciated (I've seen an error in code where you were not killing it as it should, there was an await missing)
Did we change the way we instanciate libcore in Live ?
@bumbar1 were you having this issue before 1.2.0 ? And by the way how many accounts do you have on Live ?
Thank you for your help!

[gre]

@KhalilBellakrid we're not spawning processes, we'll only ever spawn one process and keep it singleton: the internal process as shown in the screenshot. It's main role is to run the libcore and usb interaction.

Btw you can freely KILL that process , app is resilient to recreate it on the fly but that doesn't really solve the memory leak problem

[KhalilBellakrid]

Hmm I see, then we will need to analyze/profile libcore bindings ASAP (already did that with Valgrind for libcore but nothing alarming was revealed)

[gre]

actually I have an idea of what's happening. it's a classical back pressure problem:

since we can't interrupt the sync .execute() ( https://github.com/LedgerHQ/ledger-live-desktop/blob/develop/src/helpers/libcore.js#L246 ) that potentially is taking forever ( HTTP + if lot of transactions ) yet we'll timeout it in the UI side logic ( https://github.com/LedgerHQ/ledger-live-desktop/blob/develop/src/bridge/BridgeSyncContext.js#L83 ).

the problem with this approach is that the execute(), being a promise, can't be interrupted so it will just get ignored, continued in background while we'll request AGAIN the same sync N minutes later.

The answer to that will be maybe to never interrupt the sync even if it takes forever AND implement parallelization.
This current implementation is a legit workaround to do one sync at a time because we reached during the pre-release rush problems with the parallelization (some sync was never ending, probably race condition somewhere)

[KhalilBellakrid]

It's a good lead, that's why I was planning on focusing on libcore' binding and specially how you use/interact with it, so I will plan a session with you or @meriadec to discuss about it so we can find the root cause of this issue.

[bumbar1]

I have bitcoin, ethereum and zencash (horizen) currently installed, each with multiple accounts. I have about 15 accounts spread between listed currencies, and less than 100 transactions combined. I was watching memory usage and roughly on average every 30 seconds usage would go up for about 400-500 MB.

I did not have this issue prior to 1.2.0, but there was another memory issue: after running in background for a while (and synced), memory usage would skyrocket in seconds and it completely killed my machine (filled ram and swap completely).

[teams2ua]

Should be fixed with https://github.com/LedgerHQ/lib-ledger-core-node-bindings/pull/18