Spleeter: [Bug] Training crashes

[aidv]

I'm training using a dataset of 11000 audio files and after some time it just crashes.

Output

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: assertion failed: [Need value.shape >= size, got ] [858 3072 1] [512 3072 2]
         [[{{node random_crop/Assert/Assert}}]]
         [[IteratorGetNext]]
  (1) Invalid argument: assertion failed: [Need value.shape >= size, got ] [858 3072 1] [512 3072 2]
         [[{{node random_crop/Assert/Assert}}]]
         [[IteratorGetNext]]
         [[gradients/sub_grad/Shape_1/_2089]]
0 successful operations.
0 derived errors ignored.

Environment

| | |
| ----------------- | ------------------------------- |
| OS | Windows |
| Installation type | Conda |
| RAM available | 32GB |
| Hardware spec | CPU: 9900K, GPU: RTX2080 |

Why does this happen?

I've noticed it can happen at different times, such as after 680 files have been loaded or after 820 files have been loaded, so it does not seem to occur at a specific file.

[mmoussallam]

Hi @aidv

Don't know for sure what is happening but it looks like a tensor shape issue. Are some of your files mono while the rest is stereo ?

[aidv]

@mmoussallam I have a hard time thinking that any of the files are mono. I removed the files that caused the issue and it seemed to continue fine after that.

I proceeded by scaling down the dataset for testing purposes, and then a new problem occured, which I've explained here Issue #215

[iver56]

Indeed, this happened to me when I tried to train on mono audio files, even though I specified "n_channels": 1 in the json config file. Converting my audio files to stereo helped.

[aidv]

Indeed that was the problem. Using stereo files exclusively seems to be the best approach.