Core: Fritz!Box Net Monitor broken due to failing requirements

[sebk-666]

Home Assistant release with the issue:
I'm currently at 0.102.3; it started a few releases back, though

Last working Home Assistant release (if known):
unknown

Operating environment (Hass.io/Docker/Windows/etc.):
Docker on x86, official HA image

Integration:
https://www.home-assistant.io/integrations/fritzbox_netmonitor/

Description of problem:
The fritzbox_netmonitor sensor platform is currently broken in recent Docker images.
It requires a package fritzconnection==0.8.4 to be installed and HA tries to install it during startup but fails to do due to conflicts with other packages.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

sensor:
  platform: fritzbox_netmonitor
  host: 192.168.x.x

Traceback (if applicable):

2019-12-12 12:06:05 INFO (SyncWorker_2) [homeassistant.loader] Loaded fritzbox_netmonitor from homeassistant.components.fritzbox_netmonitor
2019-12-12 12:06:06 INFO (SyncWorker_6) [homeassistant.util.package] Attempting install of fritzconnection==0.8.4
2019-12-12 12:06:08 INFO (SyncWorker_2) [homeassistant.loader] Loaded fritz from homeassistant.components.fritz
2019-12-12 12:06:31 ERROR (SyncWorker_6) [homeassistant.util.package] Unable to install package fritzconnection==0.8.4: ERROR: roku 3.1 has requirement lxml<4.5.0,>=4.4.0, but you'll have lxml 4.3.4 which is incompatible.
ERROR: roku 3.1 has requirement requests<2.11,>=2.10, but you'll have requests 2.22.0 which is incompatible.
ERROR: Could not install packages due to an EnvironmentError: [Errno 39] Directory not empty: '/usr/local/lib/python3.7/site-packages/lxml-4.4.2.dist-info/'

WARNING: You are using pip version 19.3; however, version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
2019-12-12 12:06:31 ERROR (MainThread) [homeassistant.config] Platform error: sensor - Requirements for fritzbox_netmonitor not found: ['fritzconnection==0.8.4'].
2019-12-12 12:06:32 INFO (SyncWorker_6) [homeassistant.util.package] Attempting install of fritzconnection==0.8.4
2019-12-12 12:06:52 ERROR (SyncWorker_6) [homeassistant.util.package] Unable to install package fritzconnection==0.8.4: ERROR: Error checking for conflicts.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3012, in _parsed_pkg_info
    return self._pkg_info
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _pkg_info

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 567, in _warn_about_conflicts
    package_set, _dep_info = check_install_conflicts(to_install)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/check.py", line 114, in check_install_conflicts
    package_set, _ = create_package_set_from_installed()
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/check.py", line 53, in create_package_set_from_installed
    package_set[name] = PackageDetails(dist.version, dist.requires())
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3032, in _compute_dependencies
    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3014, in _parsed_pkg_info
    metadata = self.get_metadata(self.PKG_INFO)
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1420, in get_metadata
    value = self._get(path)
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1616, in _get
    with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/site-packages/lxml-4.4.2.dist-info/METADATA'
ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/site-packages/lxml-4.4.2.dist-info/RECORD'

WARNING: You are using pip version 19.3; however, version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
2019-12-12 12:06:52 ERROR (MainThread) [homeassistant.config] Platform error: device_tracker - Requirements for fritz not found: ['fritzconnection==0.8.4'].

Additional information:
none

[springstan]

@sebk-666 have you tried to update to 103.0? I am on that version and I do not have any problem with fritzbox_netmonitor so far.

[sebk-666]

I will try as soon as 0.103.1 is out.
I only tested .0 for a short moment as I lost my zwave thermostats with that. I'm expecting this fixed in .1 and will report back then...

[sebk-666]

I just tested with 0.103.1 and I'm still getting the same errors.
On 0.103.2 as well...

[springstan]

@sebk-666 I am not quite sure why but you seem to have the wrong version of roku installed. Please update it to the latest version, which is used in HA, by running pip install roku==4.0.0 inside of your venv.

[sebk-666]

@springstan, as I mentioned I'm running the official Docker image:

# docker images                                                                                                [18:53:59]
REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
docker.io/homeassistant/home-assistant   latest              1e05e466cc3b        34 hours ago        1.11 GB

[springstan]

You can either install it manually or comment out the entire fritz integration so that the requirement for roku: https://github.com/home-assistant/home-assistant/blob/84e1b3d07fb1bf3e65383286f5dd41f30498e653/requirements_all.txt#L1752-L1753 can be sucessfully installed and then you can restart with the entire fritz integration.

[sebk-666]

Isn't the whole point of running a containerized image that all the software requirements are bundled in the image?
Why isn't fritzconnection==0.8.4 already part of the image?
And why does it come with an unsuitable/outdated version of roku?

Installing additional software in a running container is kind of pointless.

I think this should be treated as a bug in the Docker image.

[sebk-666]

...besides: trying to install fritzconnection 0.8.4 manually ultimately fails because lxml 4.3.4 does not build as there is no C compiler available in the Docker container.
(On the other hand, lxml 4.4.2 seems to be installed, but apparently fritzconnection insists on the older version)

# docker exec -ti hass /bin/bash                                                                               [12:06:39]
bash-5.0# pip3 install fritzconnection==0.8.4
Collecting fritzconnection==0.8.4
  Using cached https://files.pythonhosted.org/packages/08/74/76ac2cbf5b06f8a137181ad6b26e3516e1a6f3c9c85ebfab512181428490/fritzconnection-0.8.4-py3-none-any.whl
Collecting lxml==4.3.4
  Using cached https://files.pythonhosted.org/packages/da/b5/d3e0d22649c63e92cb0902847da9ae155c1e801178ab5d272308f35f726e/lxml-4.3.4.tar.gz
Requirement already satisfied: requests==2.22.0 in /usr/local/lib/python3.7/site-packages (from fritzconnection==0.8.4) (2.22.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests==2.22.0->fritzconnection==0.8.4) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests==2.22.0->fritzconnection==0.8.4) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests==2.22.0->fritzconnection==0.8.4) (2019.11.28)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests==2.22.0->fritzconnection==0.8.4) (1.25.7)
Building wheels for collected packages: lxml
  Building wheel for lxml (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bvqkkwsa/lxml/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bvqkkwsa/lxml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-mwh4_mp8 --python-tag cp37
       cwd: /tmp/pip-install-bvqkkwsa/lxml/
  Complete output (89 lines):
  Building lxml version 4.3.4.
  Building without Cython.
  ERROR: b'/bin/sh: xslt-config: not found\n'
  ** make sure the development packages of libxml2 and libxslt are installed **

  Using build configuration of libxslt
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/sax.py -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/builder.py -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/__init__.py -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/cssselect.py -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-3.7/lxml
  creating build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/__init__.py -> build/lib.linux-x86_64-3.7/lxml/includes
  creating build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/diff.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/builder.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/__init__.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/clean.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/defs.py -> build/lib.linux-x86_64-3.7/lxml/html
  creating build/lib.linux-x86_64-3.7/lxml/isoschematron
  copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-3.7/lxml/isoschematron
  copying src/lxml/etree.h -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/etree_api.h -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/__init__.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-3.7/lxml/includes
  creating build/lib.linux-x86_64-3.7/lxml/isoschematron/resources
  creating build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/rng
  copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/rng
  creating build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl
  copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl
  copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl
  creating build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
  copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
  copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
  copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
  copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
  copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
  copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
  running build_ext
  building 'lxml.etree' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/src
  creating build/temp.linux-x86_64-3.7/src/lxml
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/usr/local/include/python3.7m -c src/lxml/etree.c -o build/temp.linux-x86_64-3.7/src/lxml/etree.o -w
  unable to execute 'gcc': No such file or directory
  Compile failed: command 'gcc' failed with exit status 1
  creating tmp
  cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitzvp3k4dn.c -o tmp/xmlXPathInitzvp3k4dn.o
  unable to execute 'cc': No such file or directory
  *********************************************************************************
  Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
  *********************************************************************************
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for lxml
  Running setup.py clean for lxml
Failed to build lxml
ERROR: Error checking for conflicts.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3012, in _parsed_pkg_info
    return self._pkg_info
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _pkg_info

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 568, in _warn_about_conflicts
    package_set, _dep_info = check_install_conflicts(to_install)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/check.py", line 114, in check_install_conflicts
    package_set, _ = create_package_set_from_installed()
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/check.py", line 53, in create_package_set_from_installed
    package_set[name] = PackageDetails(dist.version, dist.requires())
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3032, in _compute_dependencies
    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3014, in _parsed_pkg_info
    metadata = self.get_metadata(self.PKG_INFO)
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1420, in get_metadata
    value = self._get(path)
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1616, in _get
    with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/site-packages/lxml-4.4.2.dist-info/METADATA'
Installing collected packages: lxml, fritzconnection
  Found existing installation: lxml 4.4.2
ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/site-packages/lxml-4.4.2.dist-info/RECORD'