Theano: Memory gradually increasing until crash while training simple MLP on CPU

[julienroyd]

I am running this tutorial that is training a simple MLP on MNIST. I run on CPU. While monitoring my computer activity in the training loop, I can see that the memory usage is steadily increasing, until it reaches 100% and everything crashes (need to reboot).

I am on Ubuntu 16.04 and using the bleeding edge version of Theano.
There has been a similar post but the problem seemed to arise exclusively on GPU.

Is anyone experiencing the same issue on CPU?

[a-gardner1]

I also encountered this issue, except I am on Windows 7 and am using stable 0.9.0 with Keras 1.2.2 and Python 2.7. Memory usage of the python.exe process increases until the system is unusable. Reverting to Theano 0.8.2 resolves the issue.

[julienroyd]

Thanks for the reply! How exactly can you revert to an older version of Theano? Do you need to uninstall Theano first? and how to specify the version with pip install?

[a-gardner1]

Reverting is pretty easy with pip. Just use "pip install theano==[version]" to install a specific version. In my case, "pip install theano==0.8.2." It should automatically uninstall the current version.

For another reference to this issue, users of Keras have also encountered the leak and apparently determined that the cause is in Theano 0.9.0.

[lamblin]

Any specific Theano flags?
Do you have a vague idea of the rate of growth of memory usage? i.e., how long until it fills your memory, and how large is it? I'm trying to reproduce locally, and I'm not sure how long I should wait for the effects to manifest.

If you can clone the Theano repository from GitHub, and use git bisect to find the first "bad" commit between tags for the 0.8.0 and 0.9.0 releases, that would be awesome.

[lamblin]

I'm running with Python 2.7, numpy 1.11, floatX=float32, and after more than 30 minutes, top still reports 328512 (kB) in "RES" memory, which does not change.

[nouiz]

Do you use Python 3.6? I have see problems report related to numpy on
python 3.6. Make sure to update numpy to the last release if on python 3.6.
Otherwise, downgrade to 3.5. We didn't tested 3.6 yet, so don't support it
officially. Personnaly, I would not expect problem, but I got enough report
on it that I don't recommand to use numpy or Theano on python 3.6 for now.

On Fri, Apr 7, 2017 at 7:17 PM Pascal Lamblin [e-mail]
wrote:

I'm running with Python 2.7, numpy 1.11, floatX=float32, and after more
than 30 minutes, top still reports 328512 (kB) in "RES" memory, which
does not change.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Theano/Theano/issues/5810#issuecomment-292673669, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-2zqk3Lxqb2bpLL-qLKtAO3I1PBbks5rtsP1gaJpZM4MzsVn
.

[julienroyd]

I have 8GB of RAM and it gets from 3 to 8GB in a matter of seconds right after launching the script mentioned above. I am using python 2.7 and numpy 1.12.1.

For now I have downgraded to Theano 0.8.2 as @abgcsc has suggested and the problem is gone.

[nouiz]

Which Python version?

On Fri, Apr 7, 2017 at 10:54 PM Julien [e-mail] wrote:

I have 8GB of RAM and it gets from 3 to 8GB in a matter of seconds right
after launching the script mentioned above. I am using python 2.7 and numpy
1.12.1.

For now I have downgraded to Theano 0.8.2 as @abgcsc
https://github.com/abgcsc has suggested and the problem is gone.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/Theano/Theano/issues/5810#issuecomment-292690059, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-0VX1jnxNkxQa9LxpBaql__n4NUoks5rtvbfgaJpZM4MzsVn
.

[a-gardner1]

I am also using Python 2.7 and Numpy 1.12.1 with 16 GB of RAM. The leak is quite noticeable at 200-300MB per second on my machine when running the MLP tutorial.

Here is the output of git bisect:

da800d748bcd0c79d5bb1ad71b7eedb9a306afd6 is the first bad commit
commit da800d748bcd0c79d5bb1ad71b7eedb9a306afd6
Author: slefrancois [e-mail]
Date: Tue Aug 30 09:55:57 2016 -0400

port missing doc PRs to master

:040000 040000 e0eea7dfedb49fce15dc4b56709582e35c8c52cd 3a30f29020fa7543dcfa6076 5dfc812e4c85523a M doc
:040000 040000 4209ad9c6eb233767ad63649c56efe1e5ada0ba5 16256d18b363e0377f62e3f3 37d624766874d74d M theano

[hft7h11]

I am experiencing this issue on Windows 10, Theano 0.9.0, Python 2.7.11, Numpy 1.12.1

Reverting to Theano 0.8.2 has resolved the issue, but unfortunately some libraries I use depend on Theano 0.9.0 syntax

[lamblin]

@abgcsc This commit touches only documentation, and an unused configuration option. Could you try another bisect, but calling theano-cache purge at each new checkout?
I will try with numpy 1.12.1 since it seems to be a common factor.

[a-gardner1]

@lamblin Yeah, I thought the commit looked fairly benign, but I didn't want to make any assumptions. Here are the results of another bisect with a cache purge at each iteration:

4185ef0c4475086f03d5e15419a7085c87b412a6 is the first bad commit
commit 4185ef0c4475086f03d5e15419a7085c87b412a6
Author: notoraptor [e-mail]
Date: Wed Oct 26 16:10:42 2016 -0400

New update.
Code-comment removed from test_corr.
Code factorized in alt_gemm_template.c

All tests passed.

:040000 040000 f2a8681188571e260167a6613509bd88a78d50ac b6930bca1919e43b36628ddf
d14420ee4d4084da M theano

[lamblin]

Oh, it would make more sense, then.
This code falls back on NumPy's C API for dot products when we cannot directly link with a BLAS, and should not be selected when numpy has been installed with Anaconda (which provides MKL bindings), which may be why I did not manage to reproduce it.
For the moment, you could try to either:

• install numpy and other requirements through Anaconda, as well as mkl
• install a version of BLAS through your OS, and add the flags required to link with it in the config variable blas.ldflags

There's probably a missing decref somewhere in the code, I'll have a look.

[lamblin]

Can you try #5832 and confirm it fixes your case? It did for me locally.

[a-gardner1]

@lamblin I can confirm that the issue is indeed resolved on my machine with #5832. Thanks!

[aschmied]

As a quick workaround you can try adding

[global]
optimizer_excluding=GemmOptimizer

to your ~/.theanorc (%userprofile%\.theanorc on Windows). Might be helpful if upgrading is difficult.