Pipenv: Unable to create a new pipenv powered project on Windows

Created on 8 Jun 2020 · 25Comments · Source: pypa/pipenv

Issue description

When trying to set up a new project in pipenv, it is unable to find the scripts dir from the virtual environment

Expected result

Create new virtual environment and create Pipfile for project.

Actual result

$ pipenv install boto3
Creating a virtualenv for this project…
Pipfile: E:\ScriptDir\Generic\test\Pipfile
Using C:/Users/nick.perkins/AppData/Local/Programs/Python/Python37/python.exe (3.7.4) to create virtualenv…
[    ] Creating virtual environment...Already using interpreter C:\Users\nick.perkins\AppData\Local\Programs\Python\Python37\python.exe
Using base prefix 'C:\\Users\\nick.perkins\\AppData\\Local\\Programs\\Python\\Python37'
New python executable in C:\Users\nick.perkins\.virtualenvs\test-9QO8X7jI\Scripts\python.exe
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter C:/Users/nick.perkins/AppData/Local/Programs/Python/Python37/python.exe

Successfully created virtual environment!
Virtualenv location: C:\Users\nick.perkins\.virtualenvs\test-9QO8X7jI
Creating a Pipfile for this project…
Traceback (most recent call last):
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\nick.perkins\AppData\Local\Programs\Python\Python37\Scripts\pipenv.exe\__main__.py", line 9, in <module>
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\cli\command.py", line 252, in install
    site_packages=state.site_packages
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 1929, in do_install
    site_packages=site_packages,
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 615, in ensure_project
    validate=validate, skip_requirements=skip_requirements, system=system
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 320, in ensure_pipfile
    project.create_pipfile(python=python)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\project.py", line 753, in create_pipfile
    required_python = self.which("python", self.virtualenv_location)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\project.py", line 1126, in which
    result = next(iter(filter(None, (find(finder) for finder in self.finders))), None)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\project.py", line 1126, in <genexpr>
    result = next(iter(filter(None, (find(finder) for finder in self.finders))), None)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\pythonfinder\pythonfinder.py", line 134, in which
    return self.system_path.which(exe)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 483, in which
    return next(iter(f for f in filtered if f is not None), None)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 483, in <genexpr>
    return next(iter(f for f in filtered if f is not None), None)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 482, in <genexpr>
    filtered = (sub_which(self.get_path(k)) for k in self.path_order)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 438, in get_path
    raise ValueError("Path not found or generated: {0!r}".format(path))
ValueError: Path not found or generated: WindowsPath('c:/users/nick.perkins/.virtualenvs/test-9qo8x7ji/scripts')

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

pipenv install boto3

No Pipfile was createdpipenv

$ pipenv --support

Pipenv version: '2020.6.2'

Pipenv location: 'c:\\users\\nick.perkins\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\pipenv'

Python location: 'c:\\users\\nick.perkins\\appdata\\local\\programs\\python\\python37\\python.exe'

Python installations found:

3.7.4: C:\Users\nick.perkins\AppData\Local\Programs\Python\Python37\python.exe
3.7.2: C:\Users\nick.perkins\.pyenv\pyenv-win\versions\3.7.2\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.4',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.14393',
 'python_full_version': '3.7.4',
 'python_version': '3.7',
 'sys_platform': 'win32'}

System environment variables:

ACLOCAL_PATH
ALLUSERSPROFILE
APPDATA
AWS_PROFILE
CLIENTNAME
COMMONPROGRAMFILES
COMPUTERNAME
COMSPEC
CONFIG_SITE
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
DISPLAY
EXEPATH
HOME
HOMEDRIVE
HOMEPATH
HOSTNAME
INFOPATH
LANG
LOCALAPPDATA
LOGONSERVER
MANPATH
MINGW_CHOST
MINGW_PACKAGE_PREFIX
MINGW_PREFIX
MSYSTEM
MSYSTEM_CARCH
MSYSTEM_CHOST
MSYSTEM_PREFIX
NUMBER_OF_PROCESSORS
OLDPWD
ORIGINAL_PATH
ORIGINAL_TEMP
ORIGINAL_TMP
OS
PATH
PATHEXT
PKG_CONFIG_PATH
PLINK_PROTOCOL
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMFILES
PS1
PSMODULEPATH
PUBLIC
PWD
PYENV
PROGRAMDATA
PROGRAMFILES(X86)
PROGRAMW6432
SESSIONNAME
SHELL
SHLVL
SSH_ASKPASS
SYSTEMDRIVE
SYSTEMROOT
TEMP
TERM
TMP
TMPDIR
USERDNSDOMAIN
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERNAME
USERPROFILE
WINDIR
_
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

PipenvÔÇôspecific environment variables:

DebugÔÇôspecific environment variables:

PATH: C:\Users\nick.perkins\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\nick.perkins\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files\Amazon\cfn-bootstrap;C:\Program Files\Git\cmd;C:\Program Files\7-Zip;C:\Program Files\TortoiseGit\bin;C:\Program Files\Amazon\AWSCLIV2;C:\Users\nick.perkins\.pyenv\pyenv-win\bin;C:\Users\nick.perkins\.pyenv\pyenv-win\shims;C:\Users\nick.perkins\AppData\Local\Programs\Python\Python37\Scripts;C:\Users\nick.perkins\AppData\Local\Programs\Python\Python37;C:\Users\nick.perkins\AppData\Local\Microsoft\WindowsApps;C:\Users\nick.perkins\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl
SHELL: C:\Program Files\Git\usr\bin\bash.exe
LANG: en_US.UTF-8
PWD: E:/ScriptDir/Generic/test

Type Regression

Source

nickperkins

👍12 🚀1 🎉1

Most helpful comment

@Timtam this has been fixed in the upstream version which haven't been released yet.
You can install the latest version from git:
pip install -e git+https://github.com/pypa/pipenv.git

Or patch your local installation with the fixes in the pull request.

@frostming any idea when a new version will be released?

rfkortekaas on 9 Oct 2020

👍6

All 25 comments

+1 had this same problem

max-niederman on 8 Jun 2020

👍1

I've investigated this issue and it is introduced in the normalize_path function. At some point the path_order list in the pythonfinder path model contains the paths with normalized case while the get_path function gets a path without normalized case and the path cannot be found in the defaultdict due to case differences. It looks like this only happens on windows when pyenv is also installed.

The normalize_path function was already on the nomination to be refactored in vistir.

rfkortekaas on 16 Jun 2020

crmne on 2 Jul 2020

Ran into the same issue today

JonasR on 6 Jul 2020

I've investigated this issue and it is introduced in the normalize_path function. At some point the path_order list in the pythonfinder path model contains the paths with normalized case while the get_path function gets a path without normalized case and the path cannot be found in the defaultdict due to case differences. It looks like this only happens on windows when pyenv is also installed.

The normalize_path function was already on the nomination to be refactored in vistir.

is there a known workaround to this?

UPDATE: roll back to 2018.11.26 release (https://pypi.org/project/pipenv/2018.11.26/)

dennisroche on 10 Jul 2020

❤5

I've investigated this issue and it is introduced in the normalize_path function. At some point the path_order list in the pythonfinder path model contains the paths with normalized case while the get_path function gets a path without normalized case and the path cannot be found in the defaultdict due to case differences. It looks like this only happens on windows when pyenv is also installed.
The normalize_path function was already on the nomination to be refactored in vistir.

is there a known workaround to this?

UPDATE: roll back to 2018.11.26 release (https://pypi.org/project/pipenv/2018.11.26/)

This resolved my problem, thanks a bunch!

ghanshyam30 on 20 Jul 2020

👍1

It looks like this only happens on windows when pyenv is also installed.

I don't have pyenv installed

nickperkins on 29 Jul 2020

I've found that if I add a Pipfile to the target folder and then run pip install I can get a new pipenv working

nickperkins on 29 Jul 2020

update your virtualenv pipenv
pip install pipenv==2018.11.26
this works for me

prataplyf on 31 Jul 2020

👍3

That's great but that's not an upgrade

On Fri, 31 Jul 2020, 5:30 pm Ashish Pratap Singh, notifications@github.com
wrote:

update your virtualenv pipenv
pip install pipenv==2018.11.26
this works for me

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pipenv/issues/4330#issuecomment-666978731, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAELERBVBIQM6FQXXXXG6HLR6JXJRANCNFSM4NYCKWIA
.

nickperkins on 31 Jul 2020

That's great but that's not an upgrade
…
On Fri, 31 Jul 2020, 5:30 pm Ashish Pratap Singh, @.*> wrote: update your virtualenv pipenv pip install pipenv==2018.11.26 this works for me — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4330 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAELERBVBIQM6FQXXXXG6HLR6JXJRANCNFSM4NYCKWIA .

that's true.

might be some of your modules that are not updated and that not support the updated pipenv.

so when you update all modules the updated pipenv will work for you.

prataplyf on 1 Aug 2020

Just to make it clear to everyone,

pip install pipenv==2018.11.26 installs an older version of pipenv. It is NOT an upgrade.

nickperkins on 5 Aug 2020

Same issue here.

darnley on 13 Aug 2020

Same issue here, can't actually use pipenv.
Going back to the 2018.11.26 version does allow me to pip install inside my project folder.

davem99 on 15 Aug 2020

👍2

@nickperkins

It looks like there was some kind of regression after 2018.11.26
As @rfkortekaas mentioned, seems to be an issue of normalized case in the path. Pipenv is trying to search for a python executable path, but on windows it is trying to find a as_posix path in self.path_order which has paths formatted with double \\ instead of single / (as changed by as_posix)

The relevant pipenv code seems to be in: pipenv\vendor\pythonfinder\models\path.py (Line 432).
I can get the latest version of pyenv to work if i change Line 432 to the following:
if not _path and os.path.normcase(path.as_posix()) in self.path_order:

I'm not sure anything else would need to be done to make this more of a robust workaround, but it should work cross-platform since os.path.normcase does nothing on other platforms.

meichthys on 26 Aug 2020

@ken-reitz @techalchemy
I've submitted a fix for this upstream. Could we get this pulled in when/if the upstream pull request is accepted?

meichthys on 26 Aug 2020

@meichthys Can you figure out where the back-slashed paths are introduced since I searched all the path_order assignments and all are normalized by as_posix().

frostming on 27 Aug 2020

Given the complicated and varying env setup, a minimal reproducible example is of great help.

frostming on 27 Aug 2020

@frostming some path entries are setup with normalize_path from pipenv\pythonfinder\utils.py. This function uses normcase and normpath. On windows this changes forward slashes to backward slashes. Where as_posix returns the path with forward slashes. It took me a long time the track down the issue and I ended up with removing normcase from normalize_path what basically results in the same as the solution from @meichthys.

The issue only occurs when one of the _setup_* functions are called. So it looks like the path_order with the wrong formatting is coming form the new_instances created in the _setup_* functions.

rfkortekaas on 27 Aug 2020

👍2

@rfkortekaas Thanks, that is enlightening. I have addressed it in the upstream. https://github.com/sarugaku/pythonfinder/pull/89

frostming on 27 Aug 2020

🚀2

@frostming Thanks. I've tested your solution in the PR and it looks like everything is working as it should!

rfkortekaas on 27 Aug 2020

Also tested and working! 🚀

meichthys on 27 Aug 2020

Sorry to bring this up again, but I'm running into the same issues right now.

pipenv install black isort flake8 mypy --dev

Creating a virtualenv for this project…

Pipfile: F:\Privat\Skripte\pyAGE\Pipfile

Using default python from c:\users\toni\.local\pipx\venvs\pipenv\scripts\python.exe (3.8.2) to create virtualenv…


[    ] Creating virtual environment...
[=   ] Creating virtual environment...
[==  ] Creating virtual environment...
[=== ] Creating virtual environment...
[ ===] Creating virtual environment...
[  ==] Creating virtual environment...
[   =] Creating virtual environment...created virtual environment CPython3.8.2.final.0-64 in 427ms
  creator CPython3Windows(dest=C:\Users\Toni\.virtualenvs\pyAGE-Qx2pNdmj, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\Toni\AppData\Local\pypa\virtualenv)
    added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Successfully created virtual environment! 
Virtualenv location: C:\Users\Toni\.virtualenvs\pyAGE-Qx2pNdmj
Creating a Pipfile for this project…
Traceback (most recent call last):
  File "c:\users\toni\.pyenv\pyenv-win\versions\3.8.2\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\toni\.pyenv\pyenv-win\versions\3.8.2\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Toni\.local\bin\pipenv.exe\__main__.py", line 7, in <module>
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\cli\command.py", line 232, in install
    retcode = do_install(
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\core.py", line 1920, in do_install
    ensure_project(
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\core.py", line 614, in ensure_project
    ensure_pipfile(
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\core.py", line 320, in ensure_pipfile
    project.create_pipfile(python=python)
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\project.py", line 755, in create_pipfile
    required_python = self.which("python", self.virtualenv_location)
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\project.py", line 1128, in which
    result = next(iter(filter(None, (find(finder) for finder in self.finders))), None)
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\project.py", line 1128, in <genexpr>
    result = next(iter(filter(None, (find(finder) for finder in self.finders))), None)
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\vendor\pythonfinder\pythonfinder.py", line 134, in which
    return self.system_path.which(exe)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 483, in which
    return next(iter(f for f in filtered if f is not None), None)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 483, in <genexpr>
    return next(iter(f for f in filtered if f is not None), None)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 482, in <genexpr>
    filtered = (sub_which(self.get_path(k)) for k in self.path_order)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 438, in get_path
    raise ValueError("Path not found or generated: {0!r}".format(path))
ValueError: Path not found or generated: WindowsPath('c:/users/toni/.virtualenvs/pyage-qx2pndmj/scripts')

Here is the installed pipenv version:

pipx list

venvs are in C:\Users\Toni\.local\pipx\venvs
apps are exposed on your $PATH at C:\Users\Toni\.local\bin
   package pipenv 2020.8.13, Python 3.8.2
    - pipenv-resolver.exe
    - pipenv.exe

Is there no way to solve this except going back to a two-year old version by now?
Thanks.

Timtam on 9 Oct 2020

@Timtam this has been fixed in the upstream version which haven't been released yet.
You can install the latest version from git:
pip install -e git+https://github.com/pypa/pipenv.git

Or patch your local installation with the fixes in the pull request.

@frostming any idea when a new version will be released?

rfkortekaas on 9 Oct 2020

👍6

I needed to specify the egg name to pull the latest version
pip install -e git+https://github.com/pypa/pipenv.git#egg=pipenv