pipenv shell fails to find /usr/local/bin/bash in pexpect

Created on 3 Jul 2018 · 15Comments · Source: pypa/pipenv

Issue description

I can't run pipenv shell. This is different to the other issues related to this issue, since my SHELL variable is correctly configured. It seems to be something pexpect is doing wrong.

Expected result

Successfully running pipenv shell

Actual result

$ pipenv --version
pipenv, version 2018.7.1
$ echo $SHELL
/usr/local/bin/bash
$ stat $SHELL
16777220 8618578105 lrwxr-xr-x 1 keegan admin 0 30 "Jun 15 08:03:49 2018" "Jun 15 08:03:49 2018" "Jun 15 08:03:49 2018" "Jun 15 08:03:49 2018" 4194304 0 0 /usr/local/bin/bash
$ pipenv shell --verbose
Launching subshell in virtual environment…
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pipenv/cli.py", line 664, in shell
    three=three, python=python, fancy=fancy, shell_args=shell_args, pypi_mirror=pypi_mirror
  File "/usr/local/lib/python3.7/site-packages/pipenv/core.py", line 2163, in do_shell
    shell.fork_compat(*fork_args)
  File "/usr/local/lib/python3.7/site-packages/pipenv/shells.py", line 107, in fork_compat
    self.cmd, ['-i'], dimensions=(dims.lines, dims.columns),
  File "/usr/local/lib/python3.7/site-packages/pipenv/vendor/pexpect/pty_spawn.py", line 204, in __init__
    self._spawn(command, args, preexec_fn, dimensions)
  File "/usr/local/lib/python3.7/site-packages/pipenv/vendor/pexpect/pty_spawn.py", line 276, in _spawn
    'executable: %s.' % self.command)
pipenv.vendor.pexpect.exceptions.ExceptionPexpect: The command was not found or was not executable: -/usr/local/bin/bash.

Note: /usr/local/bin/bash is a symlink to /usr/local/Cellar/bash/4.4.23/bin/bash (managed by brew). Setting SHELL to that value or /bin/bash both fail.

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.7'

Other Python installations in PATH:

2.7: /usr/local/bin/python2.7
2.7: /usr/local/bin/python2.7
2.7: /usr/bin/python2.7
3.7: /usr/local/bin/python3.7m
3.7: /usr/local/bin/python3.7
2.7.15: /usr/local/bin/python
2.7.10: /usr/bin/python
2.7.15: /usr/local/bin/python2
3.7.0: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.6.0: Tue May  8 15:22:16 PDT '
                     '2018; root:xnu-4570.61.1~1/RELEASE_X86_64',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

STY
SRCPATH
MAILCHECK
LANG
XPC_FLAGS
HISTCONTROL
OLDPWD
TERM_SESSION_ID
EDITOR
COLORTERM
PERL_MB_OPT
__CF_USER_TEXT_ENCODING
LOCKPRG
ITERM_SESSION_ID
SSH_AUTH_SOCK
USER
GOPATH
PWD
HOME
WINDOW
TERM_PROGRAM
TERM_PROGRAM_VERSION
Apple_PubSub_Socket_Render
ITERM_PROFILE
PERL_MM_OPT
TMPDIR
PERL_LOCAL_LIB_ROOT
XPC_SERVICE_NAME
SHELL
TERM
PERL5LIB
COLORFGBG
SHLVL
LOGNAME
GOBIN
PATH
HISTSIZE
HISTFILESIZE
TERMCAP
_
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /Users/keegan/.cargo/bin:/usr/local/opt/go/libexec/bin:/Users/keegan/bin:/Users/keegan/google-cloud-sdk/bin:/Users/keegan/.yarn/bin:/Users/keegan/perl5/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/keegan/.cargo/bin:/usr/local/opt/go/libexec/bin:/Users/keegan/bin:/Users/keegan/google-cloud-sdk/bin:/Users/keegan/.yarn/bin:/Users/keegan/perl5/bin:/Users/keegan/go/bin:/Users/keegan/go/bin
SHELL: /usr/local/bin/bash
EDITOR: vim
LANG: en_GB.UTF-8
PWD: /Users/keegan/go/src/github.com/weaveworks/kubediff

Contents of Pipfile ('/Users/keegan/go/src/github.com/weaveworks/kubediff/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
attrs = "*"
tabulate = "*"
PyYAML = "*"

[dev-packages]

[requires]
python_version = "3.7"

Possible Bug

Source

keegancsmith

Most helpful comment

Will there be a new release containing the bug fix?

Whisprin on 17 Jul 2018

👍6

All 15 comments

Hmmm interesting. @uranusjr does this not follow symlinks currently?

techalchemy on 3 Jul 2018

It shouldn’t need to. I use Hombrew Bash myself on Mac as well, and it works correctly. Given /bin/bash also fails I’m inclided to think this is caused by something else…

@keegancsmith Did pipenv shell frun correctly before this version?

uranusjr on 3 Jul 2018

I only occasionally do python development these days, so the last time I would of used pipenv shell would of been a few months ago. I believe I used it back then, but I'm not 100% certain. I'm happy to try another version if you provide instructions. I have tried pipenv installed from both pip and brew.

keegancsmith on 3 Jul 2018

It’s fine. We switched to a new shell implementation recently, and if it worked previously, it is probably related to the new implementation.

I will try to debug it when I have access to my main Mac (which has the same Homebrew setup as yours).

uranusjr on 3 Jul 2018

I'm encountering the same issue with zsh (5.5.1) on Arch Linux. So it's neither bash nor Mac specific. It was working just a few days ago.

% pipenv --version
pipenv, version 2018.7.1
% echo $SHELL
/usr/bin/zsh
% stat $SHELL
  File: /usr/bin/zsh
  Size: 844656      Blocks: 1656       IO Block: 4096   regular file

It is still working on another machine with zsh + pipenv version 2018.6.25

Whisprin on 4 Jul 2018

@keegancsmith @Whisprin Can you run this with the Python you install Pipenv for me?

(For Homebrew, this would be /usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python by default.)

from pipenv.shells import _detect_emulator, detect_info
print(_detect_emulator())
print(detect_info())

And if detect_info returns a tuple, could you check if the second command is valid?

Example (from my Mac):

$ python
>>> from pipenv.shells import _detect_emulator, detect_info
>>> print(_detect_emulator())

>>> print(detect_info())
('bash', '/usr/local/bin/bash')
>>>

$ /usr/local/bin/bash -c help
GNU bash, version 4.4.23(1)-release (x86_64-apple-darwin17.5.0)
These shell commands are defined internally.  Type `help' to see this list.
Type `help name' to find out more about the function `name'.
Use `info bash' to find out more about the shell in general.
Use `man -k' or `info' to find out more about commands not in this list.

[tons of output… OK this command works.]

uranusjr on 4 Jul 2018

I see it is detecting a stray - prefix for the shell abs path.

$ /usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python
Python 3.7.0 (default, Jun 29 2018, 20:13:13)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pipenv.shells import _detect_emulator, detect_info
>>> print(_detect_emulator())

>>> print(detect_info())
('bash', '-/usr/local/bin/bash')
>>>
$ /usr/local/bin/bash -c help | head
GNU bash, version 4.4.23(1)-release (x86_64-apple-darwin17.5.0)
These shell commands are defined internally.  Type `help' to see this list.
Type `help name' to find out more about the function `name'.
Use `info bash' to find out more about the shell in general.
Use `man -k' or `info' to find out more about commands not in this list.

A star (*) next to a name means that the command is disabled.

 job_spec [&]                            history [-c] [-d offset] [n] or hist>
 (( expression ))                        if COMMANDS; then COMMANDS; [ elif C>

keegancsmith on 4 Jul 2018

So it seems shellingham parses the output of a ps command. Here is mine:

$ ps -ww -o pid= -o ppid= -o args=
 1480  1477 /Applications/iTerm.app/Contents/MacOS/iTerm2 --server login -fp keegan
 1482  1481 -bash
 1556  1482 screen
 1558  1557 -/usr/local/bin/bash
 1706  1558 /Applications/Emacs.app/Contents/MacOS/Emacs-x86_64-10_10 -nw
77061  1706 /usr/local/bin/aspell -a -m -B --encoding=utf-8
 1562  1557 -/usr/local/bin/bash
87033  1557 -/usr/local/bin/bash
84732  1557 -/usr/local/bin/bash
89065  1557 -/usr/local/bin/bash
80216  1557 -/usr/local/bin/bash

note I use GNU screen, if that makes a difference.

I can see in the vendored sourcecode if it finds a shell in a known list of shells, it doesn't strip - from the process path. However, it does in other cases. So you just need to handle that. I see the upstream has changed quite a bit in master, so maybe this problem is no longer present in the latest version of shellingham.

keegancsmith on 4 Jul 2018

Ah, I see the problem now. I was handling the shell path incorrectly if you chsh the login shell to point to an absolute path. I will use this information to fix it. Thanks so much for the help!

uranusjr on 4 Jul 2018

Will there be a new release containing the bug fix?

Whisprin on 17 Jul 2018

👍6

Is this not fixed yet in the latest on PyPI? Pretty egregious bug for a high-profile project:

⏵ pipenv --version
    pipenv, version 2018.10.13
⏵ pipenv shell
...
pipenv.vendor.pexpect.exceptions.ExceptionPexpect: The command was not found or was not executable: fish.

⏵ python3
>>> from pipenv.shells import _detect_emulator, detect_info
>>> detect_info()
('fish', 'fish')
^D

⏵ env PIPENV_SHELL="$SHELL" pipenv shell
Loading .env environment variables…
Launching subshell in virtual environment…

⏵  source .../dev-TrP3E4Ut/bin/activate.fish
(dev-TrP3E4Ut) ... ⏵

# worked.