Poetry: Unable to install `python-apt` package because of `DistUtilsExtra` module not found

[Kocal]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Question

Hi!

Some info before asking the question:

• Poetry 1.0.0b1
• Poetry setup a virtualenv with Python 3.7

I'm trying to install python-apt package (for using apt module in Ansible), but I'm facing this error:

➜  System git:(feat/dconf) ✗ poetry add python-apt                       
Using version ^0.7.8 for python-apt

Updating dependencies
Resolving dependencies... (0.1s)


Package operations: 1 install, 0 updates, 0 removals

  - Installing python-apt (0.7.8)

[EnvCommandError]
Command ['/home/kocal/.cache/pypoetry/virtualenvs/system-H90qh8t--py3.7/bin/python', '-m', 'pip', 'install', '--no-deps', 'python-apt==0.7.8'] errored with the following return code 1, and output: 
Collecting python-apt==0.7.8
  Using cached https://files.pythonhosted.org/packages/e5/ff/63bb64a103eda6f13364381e983c0121eeacc337a4421d6005ff7dd76741/python-apt-0.7.8.tar.bz2
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-hcb_ok5s/python-apt/setup.py", line 6, in <module>
        from DistUtilsExtra.command import *
    ModuleNotFoundError: No module named 'DistUtilsExtra'

Indeed, I don't see any distutils library in my virtualenv libraries:

➜  System git:(feat/dconf) ✗ l ~/.cache/pypoetry/virtualenvs/system-H90qh8t--py3.7/lib/python3.7/site-packages
total 88K
drwxr-xr-x 21 kocal kocal 4,0K sept.  7 07:37 .
drwxr-xr-x  3 kocal kocal 4,0K sept.  7 07:36 ..
drwxr-xr-x 18 kocal kocal 4,0K sept.  7 07:36 ansible
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:37 ansible-2.7.5.egg-info
-rw-r--r--  1 kocal kocal  126 sept.  7 07:36 easy_install.py
drwxr-xr-x  3 kocal kocal 4,0K sept.  7 07:37 jinja2
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:37 Jinja2-2.10.dist-info
drwxr-xr-x  3 kocal kocal 4,0K sept.  7 07:37 jmespath
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:37 jmespath-0.9.3.dist-info
drwxr-xr-x  3 kocal kocal 4,0K sept.  7 07:36 markupsafe
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:36 MarkupSafe-1.1.0.dist-info
drwxr-xr-x  5 kocal kocal 4,0K sept.  7 07:36 pip
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:36 pip-18.1.dist-info
drwxr-xr-x  5 kocal kocal 4,0K sept.  7 07:36 pkg_resources
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:36 pkg_resources-0.0.0.dist-info
drwxr-xr-x  4 kocal kocal 4,0K sept.  7 07:37 psutil
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:37 psutil-5.6.3.egg-info
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:36 __pycache__
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:37 PyYAML-5.1.1.egg-info
drwxr-xr-x  6 kocal kocal 4,0K sept.  7 07:36 setuptools
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:36 setuptools-40.8.0.dist-info
drwxr-xr-x  3 kocal kocal 4,0K sept.  7 07:37 yaml

But I don't find how to install distutils with Poetry.
It seems that distutils is only installable with system's package manager: sudo apt-get install python3-distutils.

I know that I've already ran this command, so distutils is installed for the global Python interpreter:

➜  System git:(feat/dconf) ✗ l /usr/lib/python3.7/distutils
total 500K
drwxr-xr-x  4 root root 4,0K juin   1 06:27 .
drwxr-xr-x 31 root root  20K juil. 12 06:33 ..
-rw-r--r--  1 root root 8,4K mars  26 08:58 archive_util.py
-rw-r--r--  1 root root  15K mars  26 08:58 bcppcompiler.py
-rw-r--r--  1 root root  47K mars  26 08:58 ccompiler.py
-rw-r--r--  1 root root  18K mars  26 08:58 cmd.py
drwxr-xr-x  3 root root 4,0K juin   1 06:27 command
-rw-r--r--  1 root root 4,8K mars  26 08:58 config.py
-rw-r--r--  1 root root 8,7K mars  26 08:58 core.py
-rw-r--r--  1 root root  17K mars  26 08:58 cygwinccompiler.py
-rw-r--r--  1 root root  139 mars  26 08:58 debug.py
-rw-r--r--  1 root root 3,5K mars  26 08:58 dep_util.py
-rw-r--r--  1 root root 8,2K mars  26 08:58 dir_util.py
-rw-r--r--  1 root root  50K mars  26 08:58 dist.py
-rw-r--r--  1 root root 3,5K mars  26 08:58 errors.py
-rw-r--r--  1 root root  11K mars  26 08:58 extension.py
-rw-r--r--  1 root root  18K mars  26 08:58 fancy_getopt.py
-rw-r--r--  1 root root  13K mars  26 08:58 filelist.py
-rw-r--r--  1 root root 8,0K mars  26 08:58 file_util.py
-rw-r--r--  1 root root  236 avril  3 07:39 __init__.py
-rw-r--r--  1 root root 2,0K mars  26 08:58 log.py
-rw-r--r--  1 root root  30K mars  26 08:58 msvc9compiler.py
-rw-r--r--  1 root root  22K mars  26 08:58 _msvccompiler.py
-rw-r--r--  1 root root  24K mars  26 08:58 msvccompiler.py
drwxr-xr-x  2 root root 4,0K juin   1 06:27 __pycache__
-rw-r--r--  1 root root  295 mars  26 08:58 README
-rw-r--r--  1 root root 7,3K mars  26 08:58 spawn.py
-rw-r--r--  1 root root  22K mars  26 08:58 sysconfig.py
-rw-r--r--  1 root root  13K mars  26 08:58 text_file.py
-rw-r--r--  1 root root  15K mars  26 08:58 unixccompiler.py
-rw-r--r--  1 root root  20K mars  26 08:58 util.py
-rw-r--r--  1 root root 5,1K mars  26 08:58 versionpredicate.py
-rw-r--r--  1 root root  13K avril  3 07:39 version.py

but I can't use it through Poetry.

Did anyone already encountered the issue? :confused:

Thanks!

[rdonkin]

I had a similar issue, starting with this error from Ansible when using a non-APT Python 3.7 in a virtualenv:

msg: Could not import python modules: apt, apt_pkg. Please install python-apt package.

Trying to install python-apt led to the same problem that you had with the error about DistUtilsExtra, but installing python-apt from PyPA is not the right solution, as it's very old, and distutils is a red herring as well.

Instead, here's what did work for Ubuntu 18.04, by installing the APT package python3-apt into the virtualenv in a rather hacky way.

I'm not using poetry for this but I am using a virtualenv and Python 3.7 (to get ansible-vault to decrypt properly), so I think this solution will work along with poetry. (BTW I like poetry but have other priorities right now.)

This is based on the very helpful solution by @acardak - that issue is locked so I thought I would comment here. I had to adjust that solution a little (version numbers in filenames) so it worked on fully updated Ubuntu 18.04.

#!/bin/bash
# Before starting, do `apt install python3.7` and 
# ensure `python3` is a symlink to that binary, 
# so that `python3 --version` reports 3.7.

# Installs python3.7 binary into virtualenv as python3
python3 -m virtualenv ~/venv-py37           
source ~/venv-py37/bin/activate

# Install python3-apt package into virtualenv, based on 
# https://github.com/ansible/ansible/issues/14468#issuecomment-459630445
# This package must be installed carefully so it works inside virtualenv 
# with Python 3.7, not the system default Python 3.6

# Download and extract python3-apt files into /tmp/python3-apt
pushd /tmp
apt-get download python3-apt
dpkg -x python3-apt_1.6.4_amd64.deb python3-apt
popd

# Install python3-apt files into virtualenv
cp -r /tmp/python3-apt/usr/lib/python3/dist-packages/* ~/venv-py37/lib/python3.7/site-packages/

# Rename shared libs inside the virtualenv packages dir
pushd ~/venv-py37/lib/python3.7/site-packages/
mv apt_pkg.cpython-36m-x86_64-linux-gnu.so apt_pkg.so
mv apt_inst.cpython-36m-x86_64-linux-gnu.so apt_inst.so
popd

[Kocal]

Hum interesting, I will give it a try when I will be back home.
Thanks! :)

[eff917]

The problem with python-apt is that the package is not updated on pypi. I was able to install it from this repo: https://salsa.debian.org/apt-team/python-apt/

[finswimmer]

I close this issue as it seems to be an upstream problem and not by poetry.

If someone not agree with it, please let me know.

[refulgent-de]

Your solution helped me alot, @rdonkin. To show my gratitude I allowed myself to port your solution into an ansible script : )

- name: Install python3-apt 
  when:
    - ansible_facts['distribution'] == 'Ubuntu'
    - ansible_facts['distribution_major_version'] == '18'
  block:

    - file:
        path: "{{ virtualenvDirectory }}/src"
        state: directory
        owner: "{{ systemUser }}"
        group: "{{ systemUser }}"
        mode: '0750'

    - name: Download python3-apt package
      shell: "apt-get download python3-apt"
      args:
        chdir: "{{ virtualenvDirectory }}/src"
        warn: false

    - name: Get filename of downloaded package
      find:
        paths: "{{ virtualenvDirectory }}/src"
        file_type: file
        patterns: python3-apt*
      register: findResult

    - fail:
        msg: Downloaded package was not found
      when: findResult is undefined or findResult.files|length == 0

    - set_fact: 
        python3AptPackageFileName: "{{ findResult.files[0].path|regex_replace('^.*/(.*)$','\\1') }}"
      when: findResult is defined

    - name: Extract package
      shell: dpkg -x {{ python3AptPackageFileName }} python3-apt
      args:
        chdir: "{{ virtualenvDirectory }}/src"

    - name: Get list of sharable objects
      find:
        paths: "{{ virtualenvDirectory }}/src/python3-apt/usr/lib/python3/dist-packages" # Path should stay stable throughout python 3.x I guess
        file_type: file
        patterns: '*.so'
      register: findResult
    - set_fact:
        python3AptShareableObjectsFilenames: "{{ python3AptShareableObjectsFilenames|default([]) + [item.path|regex_replace('^.*/(.*)$','\\1')] }}"
      with_items: "{{ findResult.files }}"

    - name: Copy package into virtualenv lib directory
      copy:
        src: "{{ virtualenvDirectory }}/src/python3-apt/usr/lib/python3/dist-packages/"
        dest: "{{ virtualenvDirectory }}/lib/{{ altInterpreterFileName }}/site-packages/"
        remote_src: yes
        owner: "{{ systemUser }}"
        group: "{{ systemUser }}"
        mode: '0750'

    - name: Rename shareable objects
      shell: "mv {{ item }} {{ item|regex_replace('(.*)\\..*\\.(.*)', '\\1.\\2') }}"
      args:
        chdir: "{{ virtualenvDirectory }}/lib/{{ altInterpreterFileName }}/site-packages/"
      with_items: "{{ python3AptShareableObjectsFilenames }}"

Greetings

[rdonkin]

@finswimmer - agree this is an upstream issue, though it's useful to have the workarounds here as they are not easy to do.

@refulgent-de - glad this was useful. I tend to do this when bootstrapping an Ansible laptop, but your Ansible code will be useful for other cases.