Molecule: master: pip install -e . fails with TypeError: expected str, bytes or os.PathLike object, not NoneType

[ssbarnea]

The problem seems to be related to recent pep517/setuptools/packaging changes but is really worrig because it totally breaks packaging.

$ pip install -e .                                                                                                                                                                                                                            [15:59:21]
Obtaining file:///Users/ssbarnea/os/molecule
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    Complete output from command /Users/ssbarnea/rdo/tripleo-ci/.tox/molecule/bin/python3.6 /Users/ssbarnea/rdo/tripleo-ci/.tox/molecule/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/tmpjmy837ev:
    running dist_info
    creating /Users/ssbarnea/os/molecule/pip-wheel-metadata/molecule.egg-info
    writing /Users/ssbarnea/os/molecule/pip-wheel-metadata/molecule.egg-info/PKG-INFO
    writing dependency_links to /Users/ssbarnea/os/molecule/pip-wheel-metadata/molecule.egg-info/dependency_links.txt
    writing entry points to /Users/ssbarnea/os/molecule/pip-wheel-metadata/molecule.egg-info/entry_points.txt
    writing requirements to /Users/ssbarnea/os/molecule/pip-wheel-metadata/molecule.egg-info/requires.txt
    writing top-level names to /Users/ssbarnea/os/molecule/pip-wheel-metadata/molecule.egg-info/top_level.txt
    writing manifest file '/Users/ssbarnea/os/molecule/pip-wheel-metadata/molecule.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file '/Users/ssbarnea/os/molecule/pip-wheel-metadata/molecule.egg-info/SOURCES.txt'
    creating '/Users/ssbarnea/os/molecule/pip-wheel-metadata/molecule.dist-info'
    Traceback (most recent call last):
      File "/Users/ssbarnea/rdo/tripleo-ci/.tox/molecule/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/Users/ssbarnea/rdo/tripleo-ci/.tox/molecule/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/Users/ssbarnea/rdo/tripleo-ci/.tox/molecule/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 69, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/Users/ssbarnea/.pyenv/versions/3.6.7/lib/python3.6/site-packages/setuptools/build_meta.py", line 113, in prepare_metadata_for_build_wheel
        _run_setup()
      File "/Users/ssbarnea/.pyenv/versions/3.6.7/lib/python3.6/site-packages/setuptools/build_meta.py", line 72, in _run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 351, in <module>
        __name__ == '__main__' and setuptools.setup(**setup_params)
      File "/Users/ssbarnea/.pyenv/versions/3.6.7/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/Users/ssbarnea/.pyenv/versions/3.6.7/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/Users/ssbarnea/.pyenv/versions/3.6.7/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/Users/ssbarnea/.pyenv/versions/3.6.7/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "setup.py", line 323, in run
        return setuptools.command.dist_info.dist_info.run(self)
      File "/Users/ssbarnea/.pyenv/versions/3.6.7/lib/python3.6/site-packages/setuptools/command/dist_info.py", line 36, in run
        bdist_wheel.egg2dist(egg_info.egg_info, dist_info_dir)
      File "/Users/ssbarnea/.pyenv/versions/3.6.7/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 443, in egg2dist
        shutil.copy(license, os.path.join(self.distinfo_dir, license_filename))
      File "/Users/ssbarnea/rdo/tripleo-ci/.tox/molecule/lib/python3.6/posixpath.py", line 80, in join
        a = os.fspath(a)
    TypeError: expected str, bytes or os.PathLike object, not NoneType

    ----------------------------------------
Command "/Users/ssbarnea/rdo/tripleo-ci/.tox/molecule/bin/python3.6 /Users/ssbarnea/rdo/tripleo-ci/.tox/molecule/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/tmpjmy837ev" failed with error code 1 in /Users/ssbarnea/os/molecule

[ssbarnea]

One side-effect of this change is that you endup with a new folder pip-wheel-metadata/ created in your repository which is not included in .gitignore., clearly not a major issue but still needs to be fixed. More important is to find out what is messing it.

[webknjaz]

I think @decentral1se is trying to gitignore that. WRT the reported problem I'll take a look. Quickfix for you should be upgrading pip and setuptools

[ssbarnea]

It seems that even linux installation is broken, look below.

Upgrading pip/setuptools/tox is not always an option, but fyi this is happening with bleeding edge pip 19.0.3 too! pip freeze does not list setuptools as a package but >>> setuptools.__version__ '40.8.0' .... I am getting insane because I do still have a day to day job.

Let's hope we will all find a way to sort it out. Also finding a way to avoid it for the users should be priority: ability to install the product on various systems, something that we need to avoid breaking at all costs (me thinking about finding ways to gate it with travis or similar).

pip 19.0.3 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)

pip install --user -e .
Obtaining file:///home/cloud-user/rdo/molecule
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Requirement already satisfied: six==1.11.0 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (1.11.0)
Requirement already satisfied: colorama==0.3.9 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (0.3.9)
Requirement already satisfied: sh==1.12.14 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (1.12.14)
Requirement already satisfied: PyYAML==3.13 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (3.13)
Requirement already satisfied: ansible>=2.5 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (2.7.10)
Requirement already satisfied: tree-format==0.1.2 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (0.1.2)
Requirement already satisfied: ansible-lint<5,>=4.0.2 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (4.1.0)
Requirement already satisfied: click==6.7 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (6.7)
Requirement already satisfied: Jinja2==2.10 in /usr/local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (2.10)
Requirement already satisfied: testinfra==1.19.0 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (1.19.0)
Requirement already satisfied: python-gilt<2,>=1.2.1 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (1.2.1)
Requirement already satisfied: psutil==5.4.6; sys_platform != "win32" and sys_platform != "cygwin" in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (5.4.6)
Requirement already satisfied: cerberus<3,>=1.2 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (1.2)
Requirement already satisfied: cookiecutter==1.6.0 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (1.6.0)
Requirement already satisfied: pexpect<5,>=4.6.0 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (4.7.0)
Requirement already satisfied: click-completion==0.3.1 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (0.3.1)
Requirement already satisfied: tabulate==0.8.2 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (0.8.2)
Requirement already satisfied: anyconfig==0.9.7 in /home/cloud-user/.local/lib/python3.6/site-packages (from molecule==2.21.dev46+g948c2d7.d20190411) (0.9.7)
Requirement already satisfied: paramiko in /home/cloud-user/.local/lib/python3.6/site-packages (from ansible>=2.5->molecule==2.21.dev46+g948c2d7.d20190411) (2.4.2)
Requirement already satisfied: cryptography in /home/cloud-user/.local/lib/python3.6/site-packages (from ansible>=2.5->molecule==2.21.dev46+g948c2d7.d20190411) (2.6.1)
Requirement already satisfied: setuptools in /home/cloud-user/.local/lib/python3.6/site-packages (from ansible>=2.5->molecule==2.21.dev46+g948c2d7.d20190411) (41.0.0)
Requirement already satisfied: ruamel.yaml in /home/cloud-user/.local/lib/python3.6/site-packages (from ansible-lint<5,>=4.0.2->molecule==2.21.dev46+g948c2d7.d20190411) (0.15.91)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib64/python3.6/site-packages (from Jinja2==2.10->molecule==2.21.dev46+g948c2d7.d20190411) (1.1.1)
Requirement already satisfied: pytest!=3.0.2 in /usr/local/lib/python3.6/site-packages (from testinfra==1.19.0->molecule==2.21.dev46+g948c2d7.d20190411) (4.3.0)
Requirement already satisfied: git-url-parse in /home/cloud-user/.local/lib/python3.6/site-packages (from python-gilt<2,>=1.2.1->molecule==2.21.dev46+g948c2d7.d20190411) (1.2.2)
Requirement already satisfied: fasteners in /home/cloud-user/.local/lib/python3.6/site-packages (from python-gilt<2,>=1.2.1->molecule==2.21.dev46+g948c2d7.d20190411) (0.14.1)
Requirement already satisfied: pbr in /home/cloud-user/.local/lib/python3.6/site-packages (from python-gilt<2,>=1.2.1->molecule==2.21.dev46+g948c2d7.d20190411) (5.1.3)
Requirement already satisfied: binaryornot>=0.2.0 in /home/cloud-user/.local/lib/python3.6/site-packages (from cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (0.4.4)
Requirement already satisfied: requests>=2.18.0 in /usr/local/lib/python3.6/site-packages (from cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (2.21.0)
Requirement already satisfied: whichcraft>=0.4.0 in /home/cloud-user/.local/lib/python3.6/site-packages (from cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (0.5.2)
Requirement already satisfied: poyo>=0.1.0 in /home/cloud-user/.local/lib/python3.6/site-packages (from cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (0.4.2)
Requirement already satisfied: jinja2-time>=0.1.0 in /home/cloud-user/.local/lib/python3.6/site-packages (from cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (0.2.0)
Requirement already satisfied: future>=0.15.2 in /home/cloud-user/.local/lib/python3.6/site-packages (from cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (0.17.1)
Requirement already satisfied: ptyprocess>=0.5 in /home/cloud-user/.local/lib/python3.6/site-packages (from pexpect<5,>=4.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (0.6.0)
Requirement already satisfied: pyasn1>=0.1.7 in /home/cloud-user/.local/lib/python3.6/site-packages (from paramiko->ansible>=2.5->molecule==2.21.dev46+g948c2d7.d20190411) (0.4.5)
Requirement already satisfied: bcrypt>=3.1.3 in /home/cloud-user/.local/lib/python3.6/site-packages (from paramiko->ansible>=2.5->molecule==2.21.dev46+g948c2d7.d20190411) (3.1.6)
Requirement already satisfied: pynacl>=1.0.1 in /home/cloud-user/.local/lib/python3.6/site-packages (from paramiko->ansible>=2.5->molecule==2.21.dev46+g948c2d7.d20190411) (1.3.0)
Requirement already satisfied: asn1crypto>=0.21.0 in /home/cloud-user/.local/lib/python3.6/site-packages (from cryptography->ansible>=2.5->molecule==2.21.dev46+g948c2d7.d20190411) (0.24.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /home/cloud-user/.local/lib/python3.6/site-packages (from cryptography->ansible>=2.5->molecule==2.21.dev46+g948c2d7.d20190411) (1.12.2)
Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/site-packages (from pytest!=3.0.2->testinfra==1.19.0->molecule==2.21.dev46+g948c2d7.d20190411) (1.3.0)
Requirement already satisfied: more-itertools>=4.0.0; python_version > "2.7" in /usr/local/lib/python3.6/site-packages (from pytest!=3.0.2->testinfra==1.19.0->molecule==2.21.dev46+g948c2d7.d20190411) (6.0.0)
Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/site-packages (from pytest!=3.0.2->testinfra==1.19.0->molecule==2.21.dev46+g948c2d7.d20190411) (1.8.0)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/site-packages (from pytest!=3.0.2->testinfra==1.19.0->molecule==2.21.dev46+g948c2d7.d20190411) (19.1.0)
Requirement already satisfied: pluggy>=0.7 in /usr/local/lib/python3.6/site-packages (from pytest!=3.0.2->testinfra==1.19.0->molecule==2.21.dev46+g948c2d7.d20190411) (0.9.0)
Requirement already satisfied: monotonic>=0.1 in /home/cloud-user/.local/lib/python3.6/site-packages (from fasteners->python-gilt<2,>=1.2.1->molecule==2.21.dev46+g948c2d7.d20190411) (1.5)
Requirement already satisfied: chardet>=3.0.2 in /usr/local/lib/python3.6/site-packages (from binaryornot>=0.2.0->cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests>=2.18.0->cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (2018.11.29)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests>=2.18.0->cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (1.24.1)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/site-packages (from requests>=2.18.0->cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (2.8)
Requirement already satisfied: arrow in /home/cloud-user/.local/lib/python3.6/site-packages (from jinja2-time>=0.1.0->cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (0.13.1)
Requirement already satisfied: pycparser in /home/cloud-user/.local/lib/python3.6/site-packages (from cffi!=1.11.3,>=1.8->cryptography->ansible>=2.5->molecule==2.21.dev46+g948c2d7.d20190411) (2.19)
Requirement already satisfied: python-dateutil in /home/cloud-user/.local/lib/python3.6/site-packages (from arrow->jinja2-time>=0.1.0->cookiecutter==1.6.0->molecule==2.21.dev46+g948c2d7.d20190411) (2.8.0)
Installing collected packages: molecule
  Running setup.py develop for molecule
    Complete output from command /usr/bin/python3.6 -c "import setuptools, tokenize;__file__='/home/cloud-user/rdo/molecule/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: option --user not recognized

    ----------------------------------------
Command "/usr/bin/python3.6 -c "import setuptools, tokenize;__file__='/home/cloud-user/rdo/molecule/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=" failed with error code 1 in /home/cloud-user/rdo/molecule/

[webknjaz]

@ssbarnea please tell me versions of your setuptools and pip both inside your virtualenv and outside (pyenv's site-packages)

Possible related problems:

• https://github.com/pypa/pip/issues/6264
• https://github.com/pypa/pip/pull/6370

[webknjaz]

Editable installs are not critical. Users mustn't install dists like this in production

[webknjaz]

pip install --user -e .

Oh, this is totally weird. Don't do this.

[webknjaz]

btw works for me. I cannot reproduce this. Is there any actual reproducer maybe in the form of a docker container?

[webknjaz]

error: option --user not recognized

this one is reproducible. but it's still something we shouldn't support in the first place.

[webknjaz]

/me :star_of_david: invokes the higher powers...

@pradyunsg come to us!

[webknjaz]

@ssbarnea please add print(setuptools.__version__) right before the last line with setup() call in setup.py and post what it prints. It's also important for debugging.

Current summary:

• don't use --editable in production
• dev env with tox works fine
• --editable + --user probably has undefined behavior and I strongly discourage using it anyway

[webknjaz]

P.S. Just pip install -e . _seems legit_ but I'm still waiting for a reproducer (Dockerfile would do) because without it — works for me + works in tox. Until that, I'm going to assume some mysterious weird environment.

[ssbarnea]

Upgrade pip to avoid it!

I was able to find an workaround for this issue, which may be problematic in some cases:

WARNING: You are using pip version 19.1, however version 19.1.1 is available.

[akash-harijan]

upgrade you pip via

curl https://bootstrap.pypa.io/get-pip.py | python -

I had same problem, and I was using pip (9.0.1) version, but when I upgraded pip to (20.3.3), it got resolved.