Spleeter: [Bug] Spleeter error running in Python (Anaconda/Windows/CUDA/Visual/Tensorflow/GPU accelerated)

Created on 15 Jan 2020 · 9Comments · Source: deezer/spleeter

Description

I tried to run spleeter from python using the sourcecode below. When running spleeter separate -i D:\SAE\Producties\PsyTrain\Kadoc.flac -o D:\SAE\ProductiesPsyTrain\5stems\ -p spleeter:5stems I get error reports and no stems are separated. Note: Spleeter used to work just fine on this install.

Step to reproduce

Installed using pip install spleeter Installed Visual/CUDA/Tensorflow/GPU
Run as .spleeter separate -i D:\SAE\Producties\PsyTrain\Kadoc.flac -o D:\SAE\ProductiesPsyTrain\5stems\ -p spleeter:5stems
Got `tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node conv2d_transpose_28/conv2d_transpose}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
```
 [[strided_slice_48/_757]]
```
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node conv2d_transpose_28/conv2d_transpose}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\sande\Anaconda3\Scripts\spleeter-script.py", line 9, in
sys.exit(entrypoint())
File "C:\Users\sande\Anaconda3\lib\site-packages\spleeter__main__.py", line 54, in entrypoint
main(sys.argv)
File "C:\Users\sande\Anaconda3\lib\site-packages\spleeter__main__.py", line 46, in main
entrypoint(arguments, params)
File "C:\Users\sande\Anaconda3\lib\site-packages\spleetercommands\separate.py", line 43, in entrypoint
synchronous=False
File "C:\Users\sande\Anaconda3\lib\site-packages\spleeter\separator.py", line 123, in separate_to_file
sources = self.separate(waveform)
File "C:\Users\sande\Anaconda3\lib\site-packages\spleeter\separator.py", line 89, in separate
'audio_id': ''})
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflowcontrib\predictor\predictor.py", line 77, in __call__
return self._session.run(fetches=self.fetch_tensors, feed_dict=feed_dict)
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 950, in run
run_metadata_ptr)
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 1350, in _do_run
run_metadata)
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node conv2d_transpose_28/conv2d_transpose (defined at \lib\site-packages\spleeter\utils\estimator.py:71) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[strided_slice_48/_757]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node conv2d_transpose_28/conv2d_transpose (defined at \lib\site-packages\spleeter\utils\estimator.py:71) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.` error

Output

Traceback (most recent call last):
  File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
    return fn(*args)
  File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node conv2d_transpose_28/conv2d_transpose}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[strided_slice_48/_757]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node conv2d_transpose_28/conv2d_transpose}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sande\Anaconda3\Scripts\spleeter-script.py", line 9, in <module>
    sys.exit(entrypoint())
  File "C:\Users\sande\Anaconda3\lib\site-packages\spleeter\__main__.py", line 54, in entrypoint
    main(sys.argv)
  File "C:\Users\sande\Anaconda3\lib\site-packages\spleeter\__main__.py", line 46, in main
    entrypoint(arguments, params)
  File "C:\Users\sande\Anaconda3\lib\site-packages\spleeter\commands\separate.py", line 43, in entrypoint
    synchronous=False
  File "C:\Users\sande\Anaconda3\lib\site-packages\spleeter\separator.py", line 123, in separate_to_file
    sources = self.separate(waveform)
  File "C:\Users\sande\Anaconda3\lib\site-packages\spleeter\separator.py", line 89, in separate
    'audio_id': ''})
  File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\contrib\predictor\predictor.py", line 77, in __call__
    return self._session.run(fetches=self.fetch_tensors, feed_dict=feed_dict)
  File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
    run_metadata_ptr)
  File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
    run_metadata)
  File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node conv2d_transpose_28/conv2d_transpose (defined at \lib\site-packages\spleeter\utils\estimator.py:71) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[strided_slice_48/_757]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node conv2d_transpose_28/conv2d_transpose (defined at \lib\site-packages\spleeter\utils\estimator.py:71) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

Original stack trace for 'conv2d_transpose_28/conv2d_transpose':
  File "\Scripts\spleeter-script.py", line 9, in <module>
    sys.exit(entrypoint())
  File "\lib\site-packages\spleeter\__main__.py", line 54, in entrypoint
    main(sys.argv)
  File "\lib\site-packages\spleeter\__main__.py", line 46, in main
    entrypoint(arguments, params)
  File "\lib\site-packages\spleeter\commands\separate.py", line 43, in entrypoint
    synchronous=False
  File "\lib\site-packages\spleeter\separator.py", line 123, in separate_to_file
    sources = self.separate(waveform)
  File "\lib\site-packages\spleeter\separator.py", line 86, in separate
    predictor = self._get_predictor()
  File "\lib\site-packages\spleeter\separator.py", line 58, in _get_predictor
    self._predictor = to_predictor(estimator)
  File "\lib\site-packages\spleeter\utils\estimator.py", line 71, in to_predictor
    return predictor.from_saved_model(latest)
  File "\lib\site-packages\tensorflow\contrib\predictor\predictor_factories.py", line 153, in from_saved_model
    config=config)
  File "\lib\site-packages\tensorflow\contrib\predictor\saved_model_predictor.py", line 153, in __init__
    loader.load(self._session, tags.split(','), export_dir)
  File "\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "\lib\site-packages\tensorflow\python\saved_model\loader_impl.py", line 269, in load
    return loader.load(sess, tags, import_scope, **saver_kwargs)
  File "\lib\site-packages\tensorflow\python\saved_model\loader_impl.py", line 422, in load
    **saver_kwargs)
  File "\lib\site-packages\tensorflow\python\saved_model\loader_impl.py", line 352, in load_graph
    meta_graph_def, import_scope=import_scope, **saver_kwargs)
  File "\lib\site-packages\tensorflow\python\training\saver.py", line 1473, in _import_meta_graph_with_return_elements
    **kwargs))
  File "\lib\site-packages\tensorflow\python\framework\meta_graph.py", line 857, in import_scoped_meta_graph_with_return_elements
    return_elements=return_elements)
  File "\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "\lib\site-packages\tensorflow\python\framework\importer.py", line 443, in import_graph_def
    _ProcessNewOps(graph)
  File "\lib\site-packages\tensorflow\python\framework\importer.py", line 236, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "\lib\site-packages\tensorflow\python\framework\ops.py", line 3751, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "\lib\site-packages\tensorflow\python\framework\ops.py", line 3751, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "\lib\site-packages\tensorflow\python\framework\ops.py", line 3641, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Environment

| | |
| ----------------- | ------------------------------- |
| OS | Windows 10 |
| Installation type | Conda / pip / |
| RAM available | 16|
| Hardware spec | NVIDIA GeForce GTX 1060 / i7-8750H / etc ... |

Additional context

I performed a clean install from GitHub before and also installed support for hardware accelleration via GPU for Spleeter, which also has worked fine before today.

bug invalid

Source

Epemaster

All 9 comments

I've had this too. Most people have. I think there are several closed issues with solutions available.

aidv on 16 Jan 2020

I've had this too. Most people have. I think there are several closed issues with solutions available.

I do not see any closed issues with solutions available, please provide link?

The problem relates to tensorflow particularly; in any case it used to work just fine before a coupe of days ago, try and explain that

Epemaster on 16 Jan 2020

@Epemaster

https://github.com/deezer/spleeter/issues?q=is%3Aissue+oom+is%3Aclosed

Try running the following command
python -m spleeter separate -i D:\SAE\Producties\PsyTrain\Kadoc.flac -o D:\SAE\ProductiesPsyTrain\5stems\ -p spleeter:5stems

aidv on 16 Jan 2020

@Epemaster

https://github.com/deezer/spleeter/issues?q=is%3Aissue+oom+is%3Aclosed

Try running the following command
python -m spleeter separate -i D:\SAE\Producties\PsyTrain\Kadoc.flac -o D:\SAE\ProductiesPsyTrain\5stems\ -p spleeter:5stems

I tried, the same error pops up (it's an error on Tensorflow somehow)

(base) C:\Users\sande>pip3 install --user --upgrade tensorflow
'pip3' is not recognized as an internal or external command,
operable program or batch file.

(base) C:\Users\sande>pip3 install --user --upgrade tensorflow # install in $HOME
'pip3' is not recognized as an internal or external command,
operable program or batch file.

(base) C:\Users\sande>docker pull tensorflow/tensorflow
'docker' is not recognized as an internal or external command,
operable program or batch file.

(base) C:\Users\sande>python -m spleeter separate -i D:\SAE\Producties\PsyTrain\Kadoc.flac -o D:\SAE\ProductiesPsyTrain\5stems\ -p spleeter:5stems
Traceback (most recent call last):
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 1356, in _do_call
return fn(*args)
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node conv2d_transpose_28/conv2d_transpose}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[strided_slice_48/_757]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node conv2d_transpose_28/conv2d_transpose}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\sande\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\sande\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\sande\Anaconda3\lib\site-packagesspleeter__main__.py", line 58, in
entrypoint()
File "C:\Users\sande\Anaconda3\lib\site-packagesspleeter__main__.py", line 54, in entrypoint
main(sys.argv)
File "C:\Users\sande\Anaconda3\lib\site-packagesspleeter__main__.py", line 46, in main
entrypoint(arguments, params)
File "C:\Users\sande\Anaconda3\lib\site-packagesspleetercommands\separate.py", line 43, in entrypoint
synchronous=False
File "C:\Users\sande\Anaconda3\lib\site-packagesspleeter\separator.py", line 123, in separate_to_file
sources = self.separate(waveform)
File "C:\Users\sande\Anaconda3\lib\site-packagesspleeter\separator.py", line 89, in separate
'audio_id': ''})
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflowcontrib\predictor\predictor.py", line 77, in __call__
return self._session.run(fetches=self.fetch_tensors, feed_dict=feed_dict)
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 950, in run
run_metadata_ptr)
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 1350, in _do_run
run_metadata)
File "C:\Users\sande\Anaconda3\lib\site-packages\tensorflow\pythonclient\session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node conv2d_transpose_28/conv2d_transpose (defined at \site-packagesspleeter\utils\estimator.py:71) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[strided_slice_48/_757]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[32,16,256,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node conv2d_transpose_28/conv2d_transpose (defined at \site-packagesspleeter\utils\estimator.py:71) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

Original stack trace for 'conv2d_transpose_28/conv2d_transpose':
File "\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "\site-packagesspleeter__main__.py", line 58, in
entrypoint()
File "\site-packagesspleeter__main__.py", line 54, in entrypoint
main(sys.argv)
File "\site-packagesspleeter__main__.py", line 46, in main
entrypoint(arguments, params)
File "\site-packagesspleetercommands\separate.py", line 43, in entrypoint
synchronous=False
File "\site-packagesspleeter\separator.py", line 123, in separate_to_file
sources = self.separate(waveform)
File "\site-packagesspleeter\separator.py", line 86, in separate
predictor = self._get_predictor()
File "\site-packagesspleeter\separator.py", line 58, in _get_predictor
self._predictor = to_predictor(estimator)
File "\site-packagesspleeter\utils\estimator.py", line 71, in to_predictor
return predictor.from_saved_model(latest)
File "\site-packages\tensorflowcontrib\predictor\predictor_factories.py", line 153, in from_saved_model
config=config)
File "\site-packages\tensorflowcontrib\predictor\saved_model_predictor.py", line 153, in __init__
loader.load(self._session, tags.split(','), export_dir)
File "\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
return func(args, *kwargs)
File "\site-packages\tensorflow\python\saved_model\loader_impl.py", line 269, in load
return loader.load(sess, tags, import_scope, *saver_kwargs)
File "\site-packages\tensorflow\python\saved_model\loader_impl.py", line 422, in load
*saver_kwargs)
File "\site-packages\tensorflow\python\saved_model\loader_impl.py", line 352, in load_graph
meta_graph_def, import_scope=import_scope, *saver_kwargs)
File "\site-packages\tensorflow\python\training\saver.py", line 1473, in _import_meta_graph_with_return_elements
*kwargs))
File "\site-packages\tensorflow\python\framework\meta_graph.py", line 857, in import_scoped_meta_graph_with_return_elements
return_elements=return_elements)
File "\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "\site-packages\tensorflow\python\framework\importer.py", line 443, in import_graph_def
_ProcessNewOps(graph)
File "\site-packages\tensorflow\python\framework\importer.py", line 236, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "\site-packages\tensorflow\python\framework\ops.py", line 3751, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "\site-packages\tensorflow\python\framework\ops.py", line 3751, in
for c_op in c_api_util.new_tf_operations(self)
File "\site-packages\tensorflow\python\framework\ops.py", line 3641, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()

(base) C:\Users\sande>

Epemaster on 16 Jan 2020

What version of CUDA do you have installed?

aidv on 16 Jan 2020

Hi @Epemaster

An OOM error like yours indicate that the file you're processing is too large to fit into your memory. You may want to split your file into smaller pieces of, say 1min each and process them separately.

@aidv the python -m spleeter trick is only a solution for the unknown spleeter command issue as explained in the FAQ. it won't solve other types of problems

mmoussallam on 17 Jan 2020

👍1

@mmoussallam I've had so many errors that I just brute-force the different fixes.

Check CUDA version
Check FFMPEG installation
python -m
Split audio into pieces

aidv on 18 Jan 2020

Hi @Epemaster

An OOM error like yours indicate that the file you're processing is too large to fit into your memory. You may want to split your file into smaller pieces of, say 1min each and process them separately.

@aidv the python -m spleeter trick is only a solution for the unknown spleeter command issue as explained in the FAQ. it won't solve other types of problems

Hi @mmoussallam

The software worked perfectly fine after installing all necessary software components during December with files of similar size as current ones; this can therefore not be the issue causing the error. My laptop packs 16GB memory, 6GB RAM, 6 GB GPU memory.

Epemaster on 18 Jan 2020

@mmoussallam I've had so many errors that I just brute-force the different fixes.

Check CUDA version

Check FFMPEG installation

python -m

Split audio into pieces