Dietpi: [Solved] Bazarr not working anymore. Probably needs gevent but cannot be installed from python pip

[garret]

I have DietPi v6.19.7 installed on a Raspberry Pi 3B. Here I have had Bazarr up and running without any issues. It was installed by giving these commands on a fresh install:

apt-get install git
cd /mnt/dietpi_userdata/
git clone https://github.com/morpheus65535/bazarr.git
python /mnt/dietpi_userdata/bazarr/bazarr.py & 

Recently I have noticed that Bazarr was not working anymore because I could not access to it via browser.
So when giving the last command from above I get this error:

root@transmission:~# Bazarr starting...
2019-01-22 16:09:53,400 - root                             (76f7a8c0) :  ERROR (main:53) - BAZARR require gevent Python module to be installed using pip.
Bazarr exited.
^C
[1]+  Done                    python /mnt/dietpi_userdata/bazarr/bazarr.py
root@transmission:~#

I installed the packages _git-core_ and _python-pip_ through apt-get as suggested by Bazarr wiki. Later, I gave the command to install pip but this gives me an error. Probably this is why Bazarr is not working anymore?!

root@transmission:~# pip install gevent
Collecting gevent
  Using cached https://files.pythonhosted.org/packages/ed/27/6c49b70808f569b66ec7fac2e78f076e9b204db9cf5768740cff3d5a07ae/gevent-1.4.0.tar.gz
  Ignoring cffi: markers 'sys_platform == "win32" and platform_python_implementation == "CPython"' don't match your environment
Collecting greenlet>=0.4.14; platform_python_implementation == "CPython" (from gevent)
  Using cached https://files.pythonhosted.org/packages/f8/e8/b30ae23b45f69aa3f024b46064c0ac8e5fcb4f22ace0dca8d6f9c8bbe5e7/greenlet-0.4.15.tar.gz
Building wheels for collected packages: gevent, greenlet
  Running setup.py bdist_wheel for gevent ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-xDLbDB/gevent/setup.py';f=getattr(tokeniz  e, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpToyvuNpip-whe  el- --python-tag cp27:
  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'cffi_modules'
    warnings.warn(msg)
  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'
    warnings.warn(msg)
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for gevent
  Running setup.py clean for gevent
  Running setup.py bdist_wheel for greenlet ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-xDLbDB/greenlet/setup.py';f=getattr(token  ize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp6iWSxvpip-w  heel- --python-tag cp27:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for greenlet
  Running setup.py clean for greenlet
Failed to build gevent greenlet
Installing collected packages: greenlet, gevent
  Running setup.py install for greenlet ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-xDLbDB/greenlet/setup.py';f=getattr(tok  enize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-DjgVyd  -record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'greenlet' extension
    creating build
    creating build/temp.linux-armv7l-2.7
    arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdeb  ug-prefix-map=/build/python2.7-Ub3vap/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c   greenlet.c -o build/temp.linux-armv7l-2.7/greenlet.o
    In file included from greenlet.c:5:0:
    greenlet.h:8:20: fatal error: Python.h: No such file or directory
     #include <Python.h>
                        ^
    compilation terminated.
    error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-xDLbDB/greenlet/setup.py';f=getattr(tokenize, 'open', open)(__f  ile__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-DjgVyd-record/install-record.t  xt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-xDLbDB/greenlet/
root@transmission:~#

Do you have any idea why I get this error?

I recently reinstalled Home Assistant from the optimized software because it got an update. I don't know if that screwed something in the python installation that affects Bazarr (Home Assistant works ok instead).

[MichaIng]

I guess it is due to two different Python versions, 2.7 and 3.

~Just for a quick test, could you try to run the bazaar script with python3 command?~
Otherwise we need to make a deeper look. Generally Python2 and 3 should work fine beside each other as long as you start each script with the correct version.

---

Further check:

• Home assistant installs pyenv to install Python 3.6.X on a per-user basis, so only for the homeassistant user, if I get it right. This should not interfere with a globally installed Python version, e.g. via apt install python.
• Generally python and pip versions need to match, although python-pip depends on/installs the required Python version automatically. So if you install apt install python-pip, this should pull on Stretch Python2.7, apt install python3-pip pulls Python3 instead.
• Reading Bazarr install instructions: https://github.com/morpheus65535/bazarr/wiki/Installation-Running-from-Source
  
  
  
  • It requires Python2.7 and the mentioned commands will install it. pip install gevent is mentioned there as well.
  
  
  • The way HA installs Python3 via pyenv should not mix things up, but who knows.
  
  

To verify globally available python versions and compare to matching pip version as well:

• which -a python && readlink -f $(which python)
• which -a pip && readlink -f $(which pip)

---

Trying to replicate:

• Installing Bazarr according to official instructions works fine:

Collecting gevent
  Downloading https://files.pythonhosted.org/packages/e6/e9/3a693414f706e12abe60554cd73c5ae8f848b182ae58018f93d86c9eb418/gevent-1.4.0-cp27-cp27mu-manylinux1_x86                                                                             _64.whl (5.0MB)
    100% |████████████████████████████████| 5.0MB 35kB/s
Collecting greenlet>=0.4.14; platform_python_implementation == "CPython" (from gevent)
  Downloading https://files.pythonhosted.org/packages/8b/6e/f2d25875713ad0885c8d3c69269697406652e6f64e1a6bd8264f7a609327/greenlet-0.4.15-cp27-cp27mu-manylinux1_                                                                             x86_64.whl
Installing collected packages: greenlet, gevent
Successfully installed gevent-1.4.0 greenlet-0.4.15

• Installing HA on top...
  
  
  
  • Hmm, one of the APT package dependencies we install pull Python3 globally 🤔. This should not be required, respectively renders the pyenv thing somehow useless... Python 3.5.3 as current on Stretch repo, then via pyenv 3.6.3 again.
  
  
  • libglib2.0-dev > python3: https://packages.debian.org/stretch/libglib2.0-dev
  
  
• However python bazarr.py still works fine, not affected by global python3 install, not pyenv install from HA.
• Unable to replicate issue on fresh install.

---

Bazarr requiring gevent (and for this python-pip) is new, changed in Dezember last year: https://github.com/morpheus65535/bazarr/wiki/Installation-Running-from-Source/_compare/9496ecd5d4ead530da24e4dd6b901101be8328b2...af7e0eb46c7e781570408c40f1a19dbbc46839c3

• However uninstalling gevent + greenlet via pip and reinstalling works fine as well 🤔.

[MichaIng]

Hmm difference in install output/steps is:

Ignoring cffi: markers 'sys_platform == "win32" and platform_python_implementation == "CPython"' don't match your environment
...
Building wheels for collected packages: gevent, greenlet
  Running setup.py bdist_wheel for gevent ... error

• The cffi thing should be not related, but the wheels building is the one that fails first. Not sure what it is, but this is what needs to be checked.
• https://stackoverflow.com/questions/35169608/in-which-case-pip-install-building-wheel
• Seems to be pip issue that builds wheels sometimes wrong. But I don't understand all of the good enough.

@garret
Please try the following, which includes two solution attempts from different sources:

pip --no-cache-dir install greenlet
pip --no-cache-dir install gevent

• So install the dependency package first, gevent second separately.
• And don't use cache, but download them freshly.

[garret]

Thank you MichaIng for the support.

Trying the first command you told me still gives me an error:

root@transmission:~# pip --no-cache-dir install greenlet
Collecting greenlet
  Downloading https://files.pythonhosted.org/packages/f8/e8/b30ae23b45f69aa3f024b46064c0ac8e5fcb4f22ace0dca8d6f9c8bbe5e7/greenlet-0.4.15.tar.gz (59kB)
    100% |████████████████████████████████| 61kB 1.5MB/s
Installing collected packages: greenlet
  Running setup.py install for greenlet ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-RGAYZa/greenlet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-fGtLRN-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'greenlet' extension
    creating build
    creating build/temp.linux-armv7l-2.7
    arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-Ub3vap/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c greenlet.c -o build/temp.linux-armv7l-2.7/greenlet.o
    In file included from greenlet.c:5:0:
    greenlet.h:8:20: fatal error: Python.h: No such file or directory
     #include <Python.h>
                        ^
    compilation terminated.
    error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-RGAYZa/greenlet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-fGtLRN-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-RGAYZa/greenlet/
root@transmission:~#

The same is valid for the second command:

root@transmission:~# pip --no-cache-dir install gevent
Collecting gevent
  Downloading https://files.pythonhosted.org/packages/ed/27/6c49b70808f569b66ec7fac2e78f076e9b204db9cf5768740cff3d5a07ae/gevent-1.4.0.tar.gz (5.2MB)
    100% |████████████████████████████████| 5.2MB 8.0MB/s
  Ignoring cffi: markers 'sys_platform == "win32" and platform_python_implementation == "CPython"' don't match your environment
Collecting greenlet>=0.4.14; platform_python_implementation == "CPython" (from gevent)
  Downloading https://files.pythonhosted.org/packages/f8/e8/b30ae23b45f69aa3f024b46064c0ac8e5fcb4f22ace0dca8d6f9c8bbe5e7/greenlet-0.4.15.tar.gz (59kB)
    100% |████████████████████████████████| 61kB 7.7MB/s
Installing collected packages: greenlet, gevent
  Running setup.py install for greenlet ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-nA8aYN/greenlet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-HHwonb-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'greenlet' extension
    creating build
    creating build/temp.linux-armv7l-2.7
    arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-Ub3vap/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c greenlet.c -o build/temp.linux-armv7l-2.7/greenlet.o
    In file included from greenlet.c:5:0:
    greenlet.h:8:20: fatal error: Python.h: No such file or directory
     #include <Python.h>
                        ^
    compilation terminated.
    error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-nA8aYN/greenlet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-HHwonb-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-nA8aYN/greenlet/
root@transmission:~#

This is the output of the commands that you suggested earlier instead:

root@transmission:~# which -a python && readlink -f $(which python)
/usr/bin/python
/usr/bin/python2.7
root@transmission:~# which -a pip && readlink -f $(which pip)
/usr/bin/pip
/usr/bin/pip
root@transmission:~# 

I tried to use python3 as command to start Bazarr but still unsuccessful:

root@transmission:~# python3 /mnt/dietpi_userdata/bazarr/bazarr.py &
[1] 10953
root@transmission:~#   File "/mnt/dietpi_userdata/bazarr/bazarr.py", line 15
    print 'bazarr.py -h --no-update --config <config_directory>'
                                                               ^
SyntaxError: Missing parentheses in call to 'print'
^C
[1]+  Exit 1                  python3 /mnt/dietpi_userdata/bazarr/bazarr.py
root@transmission:~#

Finally, I tired to clone again Bazarr in another new folder and start it but got the same issues as above. Thus it seems related to the python installation.

On my dietpi, apart from home assistant, I also have installed through the optimized software: docker, sonarr, radarr, jackett and certbot.

[MichaIng]

@garret
Hmm: greenlet.h:8:20: fatal error: Python.h: No such file or directory - https://github.com/python-greenlet/greenlet/issues/103

• That python-dev is required is what I find in other sources as well, but on my test install, no python-dev is installed and it worked well.
• Your Python version with DietPi v6.19.7 should be far beyond 2.7.3, 2.7.13 more precisely, as can be seen from your logs.

Try to reinstall the python packages:
apt install --reinstall python python2.7 python2.7-minimal python-pip python-pip-whl

If it does still not work, just to sort it out, try to install python-dev, even that it should be not required:
apt install python-dev

A file system check and reboot to sort disk corruption out, also checking dmesg output for any disk corruption messages:

dmesg # Check output for red error messages related to disk I/O or emmc
> /forcefsck # Create fsck flag for next reboot
reboot # Which runs fsck during early boot stage, which can take a while

And if everything does not help, it is perhaps an ARM or Raspbian related issue, since I ran tests on x86_64 VM.

[garret]

Regarding the versions of python, I have run an _apt-get update and upgrade_ and there are no new versions. So I guess armhf has older versions :/

I re-installed the packages you were mentioning (_apt install --reinstall python python2.7 python2.7-minimal python-pip python-pip-whl_) but still got the same errors when trying to install the two pip applications.

So I finally gave _apt-get install python-dev_ and that solved the problem!

_greenlet_ and _gevent_ got installed without any error and now Bazarr starts without any issue!

Before I was running my tests on a dietpi in virtualbox but have stopped because I have observed that I can never reproduce the same errors. It would be nice to try to emulate the raspberry pi on virtualbox, but do not know if it is possible.

In any case, thank you very much @MichaIng, you are very precious resource for dietpi!

I am closing and marking as solved 👍

[MichaIng]

@garret
Great that it works now.

Okay, seems python-pip in many cases requires python-dev (we also install it with out python-pip install: https://github.com/Fourdee/DietPi/blob/master/dietpi/dietpi-software#L6842), but this is as well dependent on architecture and/or distro, e.g, Raspbian vs Debian.

• [x] Perhaps we should contribute this experience with the Bazarr devs, to add python-dev as a possible requirement to the Wiki install docs: https://github.com/morpheus65535/bazarr/issues/282