Keras: Cannot allocate memory when compiling a CNN.

[KeironO]

When I try to compile a CNN, I get the following error.

Problem occurred during compilation with the command line below:
/usr/bin/g++ -shared -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -march=corei7-avx -mcx16 -msahf -mno-movbe -maes -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mavx -mno-avx2 -msse4.2 -msse4.1 -mno-lzcnt -mno-rtm -mno-hle -mno-rdrnd -mno-f16c -mno-fsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mxsave -mxsaveopt --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=6144 -mtune=corei7-avx -D NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -fPIC -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/theano/gof -fvisibility=hidden -o /home/keo7/.theano/compiledir_Linux-3.16--generic-x86_64-with-LinuxMint-17-qiana-x86_64-2.7.6-64/tmpoLIylz/e157ab2f3af867deb8b752fbf03a772b.so /home/keo7/.theano/compiledir_Linux-3.16--generic-x86_64-with-LinuxMint-17-qiana-x86_64-2.7.6-64/tmpoLIylz/mod.cpp -L/usr/lib -lpython2.7
ERROR (theano.gof.cmodule): [Errno 12] Cannot allocate memory
Traceback (most recent call last):
  File "all_test_run.py", line 92, in <module>
    conv_net = create_conv_net(number_classes)
  File "all_test_run.py", line 74, in create_conv_net
    conv_net.compile(loss="binary_crossentropy", optimizer=stochastic_gradient_descent)
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 399, in compile
    allow_input_downcast=True, mode=theano_mode)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 308, in function
    output_keys=output_keys)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 526, in pfunc
    output_keys=output_keys)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1778, in orig_function
    defaults)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1642, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 690, in make_thunk
    storage_map=storage_map)[:3]
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/vm.py", line 1037, in make_all
    no_recycling))
  File "/usr/local/lib/python2.7/dist-packages/theano/tensor/elemwise.py", line 815, in make_thunk
    compute_map, no_recycling)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 1154, in make_thunk
    compute_map, no_recycling)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 918, in make_thunk
    no_recycling)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 836, in make_c_thunk
    output_storage=node_output_storage)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1209, in make_thunk
    keep_lock=keep_lock)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1147, in __compile__
    keep_lock=keep_lock)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1604, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 1174, in module_from_key
    module = lnk.compile_cmodule(location)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1515, in compile_cmodule
    preargs=preargs)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 2164, in compile_str
    p_out = output_subprocess_Popen(cmd)
  File "/usr/local/lib/python2.7/dist-packages/theano/misc/windows.py", line 75, in output_subprocess_Popen
    p = subprocess_Popen(command, **params)
  File "/usr/local/lib/python2.7/dist-packages/theano/misc/windows.py", line 36, in subprocess_Popen
    proc = subprocess.Popen(command, startupinfo=startupinfo, **params)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1223, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Any help would be greatly appreciated!

[fchollet]

If on GPU (which is almost certainly the case): your network simply doesn't fit on your GPU memory. You can either use a smaller network or buy a bigger GPU.

[nouiz]

The problem of during the compilation of Theano function. It I is a CPU
memory problem. For some complicated reason, maybe compiling the Theano
functions before loading data in ram would solve this problem or more CPU
RAM.
Le 3 nov. 2015 00:27, "François Chollet" [email protected] a
écrit :

If on GPU (which is almost certainly the case): your network simply
doesn't fit on your GPU memory. You can either use a smaller network or buy
a bigger GPU.

—
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/929#issuecomment-153245070.

[bfarzin]

I got the same error, but found that my problem came from not having enough RAM on the machine to complete the compile. On Ubuntu I followed these instructions to add swap space and I was then able to compile and run on GPU.

[hkmztrk]

If on GPU (which is almost certainly the case): your network simply doesn't fit on your GPU memory. You can either use a smaller network or buy a bigger GPU.

I'm also on GPU and training a CNN-based model. It works well with data of 30K samples, but I get the memory allocation error with data of 120K samples. Number of the parameters still the same, and even if I make my network smaller, the problem still continues.