Spleeter: [Bug] Nothing happens when "docker run .." (trap invalid opcode)

[fractalf]

OK, I found this great software today and tried it out right away on my laptop, awesome!

But now I was trying on my stationary pc (with same setup) and it doesn't run.

-> % pwd
# /home/alf/tmp
-> % ls -l .     
# total 4
# drwxrwxr-x 2 alf alf 4096 Apr  4 21:03 output
-> % ls -l output 
# total 2656
# -rw-rw-r-- 1 alf alf 2717920 Mar 17  2018 test.m4a
-> % docker run -v $(pwd)/output:/output researchdeezer/spleeter separate -i /output/test.m4a -o /output
# <nothing>
-> % 

This is exactly what worked on my laptop. It downloaded the model an split the audio.

I found a difference in the kernel log though..

-> % tail  /var/log/kern.log 
# ...
# Apr  4 21:03:45 tower kernel: [  552.800686] traps: spleeter[4636] trap invalid opcode ip:7f40e2481d99 sp:7fffea3652a0 error:0 in libtensorflow_framework.so.1[7f40e1d82000+1912000]
# ...

This was not present on my laptop when running the same command.

Some system info

-> % docker --version                                                                                   
# Docker version 19.03.8, build afacb8b7f0
-> % uname -a
# Linux tower 5.3.0-42-generic #34~18.04.1-Ubuntu SMP Fri Feb 28 13:42:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
-> % lsb_release -d    
# Description:  Linux Mint 19.3 Tricia

The only difference here is that on my laptop I'm running docker version 19.03.2 (but I didn't test rolling back, cause downgrading docker is a hassle. Betting this isn't the problem).

So, any idea why it's working on my laptop but not my stationary pc?

Pasting this mess last..

-> % inxi -Fxz
System:    Host: tower Kernel: 5.3.0-42-generic x86_64 bits: 64 compiler: gcc v: 7.4.0 Desktop: Cinnamon 4.4.8 
           Distro: Linux Mint 19.3 Tricia base: Ubuntu 18.04 bionic 
Machine:   Type: Desktop Mobo: Gigabyte model: X58A-UD5 v: x.x serial: <filter> BIOS: Award v: F5 date: 03/11/2010 
CPU:       Topology: Quad Core model: Intel Core i7 920 bits: 64 type: MT MCP arch: Nehalem rev: 5 L2 cache: 8192 KiB 
           flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 43178 
           Speed: 1619 MHz min/max: 1596/2661 MHz Core speeds (MHz): 1: 1619 2: 1620 3: 1619 4: 1619 5: 1619 6: 1620 7: 1619 
           8: 1619 
Graphics:  Device-1: Advanced Micro Devices [AMD/ATI] RV730 PRO [Radeon HD 4650] vendor: XFX Pine driver: radeon v: kernel 
           bus ID: 03:00.0 
           Display: x11 server: X.Org 1.19.6 driver: ati,radeon unloaded: fbdev,modesetting,vesa 
           resolution: 1920x1200~60Hz, 1920x1200~60Hz 
           OpenGL: renderer: AMD RV730 (DRM 2.50.0 / 5.3.0-42-generic LLVM 9.0.0) v: 3.3 Mesa 19.2.8 direct render: Yes 
Audio:     Device-1: Intel 82801JI HD Audio vendor: Gigabyte GA-EP45-DS5/GA-EG45M-DS2H driver: snd_hda_intel v: kernel 
           bus ID: 00:1b.0 
           Device-2: Advanced Micro Devices [AMD/ATI] RV710/730 HDMI Audio [Radeon HD 4000 series] vendor: XFX Pine 
           driver: snd_hda_intel v: kernel bus ID: 03:00.1 
           Device-3: N/A type: USB driver: snd-usb-audio bus ID: 4-2:2 
           Sound Server: ALSA v: k5.3.0-42-generic 
Network:   Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet vendor: Gigabyte driver: r8169 v: kernel 
           port: ae00 bus ID: 0a:00.0 
           IF: enp10s0 state: down mac: <filter> 
           Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet vendor: Gigabyte driver: r8169 v: kernel 
           port: 9e00 bus ID: 0b:00.0 
           IF: enp11s0 state: down mac: <filter> 
           Device-3: ASUSTek type: USB driver: rt2800usb bus ID: 1-6:4 
           IF: wlx2c4d5461e306 state: up mac: <filter> 
           IF-ID-1: docker0 state: down mac: <filter> 
Drives:    Local Storage: total: 3.99 TiB used: 2.18 TiB (54.6%) 
           ID-1: /dev/sda vendor: OCZ model: AGILITY4 size: 119.24 GiB 
           ID-2: /dev/sdb vendor: Seagate model: ST4000DM004-2CV104 size: 3.64 TiB 
           ID-3: /dev/sdc vendor: Samsung model: SSD 850 PRO 256GB size: 238.47 GiB 
Partition: ID-1: / size: 58.42 GiB used: 18.96 GiB (32.5%) fs: ext4 dev: /dev/sdc1 
           ID-2: /home size: 45.58 GiB used: 37.64 GiB (82.6%) fs: ext4 dev: /dev/sdc4 
           ID-3: swap-1 size: 7.45 GiB used: 0 KiB (0.0%) fs: swap dev: /dev/sdc3 
Sensors:   System Temperatures: cpu: 36.0 C mobo: N/A gpu: radeon temp: 45 C 
           Fan Speeds (RPM): N/A 
Info:      Processes: 274 Uptime: 20m Memory: 19.55 GiB used: 1.64 GiB (8.4%) Init: systemd runlevel: 5 Compilers: gcc: 7.5.0 
           Shell: zsh v: 5.4.2 inxi: 3.0.32 

[mmoussallam]

Hi @fractalf

Not easy for us to help you on that :/ the kernel log suggest a rather low-level issue with tensorflow. We'll update the docker images soon with a newer version of TF, hopefully it will solve this for you.

[fractalf]

Not easy for us to help you on that :/ the kernel log suggest a rather low-level issue with tensorflow. We'll update the docker images soon with a newer version of TF, hopefully it will solve this for you.

Ok, thanks @mmoussallam. What about internal spleeter logs? Maybe you could add an --verbose for debug logging etc? For all I know it might not be the kernel, thats just one of the logs I could find a diff in

[romi1502]

Hi @fractalf,
There is actually a --verbose option in spleeter that will output all tensorflow logs. It may help resoliving your issue.

[jbibro]

Hi @fractalf,
I had a similar issue on my side and I solved it by assigning more memory to docker container.

[xavriley]

Same here - using the --verbose flag for spleeter and running docker stats in a separate terminal confirmed I was hitting 100% of available memory, causing the process to terminate early. For reference it took around 10Gb of memory on a 2019 MBP for an 8 minute wav file. YMMV

[fractalf]

@jbibro Cheers man, will test this out. I have a total of 22GB mem on my pc, so probably docker isn't greedy enough