Pipenv: `pipenv run` does *not* ensure required packages are installed

[halloleo]

In a directory without a virtual environment the command

pipenv run python myprog.py

fails with

Traceback (most recent call last):
  File "listmycmds.py", line 19, in <module>
    import argh
ModuleNotFoundError: No module named 'argh'

although the module argh is in requirements.txt.

When I run

pipenv install

and then

pipenv run python myprog.py

everything is fine.

I expect - following the documentation - that pipenv run ... installs the required packages, but it doesn't.

Notes

1. Running Python version

• Pipenv is installed into the python 3.5 which is located at /usr/bin/python3.
• The environments I geenrate are using Python3.6 which is located at /usr/local/bin/python3.6.

2. python -m pipenv.help output

Here the output from after I ran my command:

<details><summary>$ python -m pipenv.help output</summary>

Pipenv version: `'11.10.1'`

Pipenv location: `'/usr/local/lib/python3.5/site-packages/pipenv'`

Python location: `'/usr/bin/python3'`

Other Python installations in `PATH`:

  - `2.6`: `/usr/bin/python2.6`
  - `2.6`: `/usr/bin/python2.6`
  - `2.7`: `/usr/local/bin/python2.7`
  - `2.7`: `/usr/local/bin/python2.7`
  - `3.4`: `/usr/local/bin/python3.4m`
  - `3.4`: `/usr/local/bin/python3.4`
  - `3.5`: `/usr/local/bin/python3.5m`
  - `3.5`: `/usr/local/bin/python3.5`
  - `3.6`: `/usr/local/bin/python3.6m`
  - `3.6`: `/usr/local/bin/python3.6`

  - `2.6.6`: `/usr/bin/python`
  - `2.6.6`: `/usr/bin/python2`
  - `3.6.4`: `/home/broleo/bin/python3`
  - `3.5.2`: `/usr/local/bin/python3`
  - `3.5.2`: `/usr/bin/python3`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.5.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.1.12-94.5.7.el6uek.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#2 SMP Thu Jul 20 18:39:04 PDT 2017',
 'python_full_version': '3.5.2',
 'python_version': '3.5',
 'sys_platform': 'linux'}
```

System environment variables:

  - `PYTHONDONTWRITEBYTECODE`
  - `PATH`
  - `SSH_CLIENT`
  - `EMACSEXE`
  - `HOME`
  - `INSIDE_EMACS`
  - `SHELL`
  - `TMUX`
  - `PROMPT_COMMAND`
  - `ORACLE_HOME`
  - `SSH_TTY`
  - `LS_COLORS`
  - `LANG`
  - `PIPENV_VENV_IN_PROJECT`
  - `TERM`
  - `QTINC`
  - `PIP_PYTHON_PATH`
  - `COLUMNS`
  - `HTTPS_PROXY`
  - `HOSTNAME`
  - `_`
  - `http_proxy`
  - `TMUX_PANE`
  - `HISTCONTROL`
  - `https_proxy`
  - `BROWSER`
  - `SSH_CONNECTION`
  - `LESSHISTFILE`
  - `QTDIR`
  - `PS1`
  - `SHLVL`
  - `PIPENV_DEFAULT_PYTHON_VERSION`
  - `GIT_EDITOR`
  - `HISTTIMEFORMAT`
  - `USER`
  - `DISPLAY`
  - `LOGNAME`
  - `HTTP_PROXY`
  - `PWD`
  - `MAIL`
  - `WORKON_HOME`
  - `QTLIB`
  - `CVS_RSH`
  - `MYCMDSPATH`
  - `VENV`
  - `LESSOPEN`
  - `LD_LIBRARY_PATH`
  - `HISTSIZE`
  - `G_BROKEN_FILENAMES`
  - `TERMCAP`

Pipenv–specific environment variables:

 - `PIPENV_VENV_IN_PROJECT`: `yes`
 - `PIPENV_DEFAULT_PYTHON_VERSION`: `3.6`

Debug–specific environment variables:

  - `PATH`: `/home/broleo/bin:/working/software/bin::/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/u01/app/oracle/product/11.2.0/bin:/home/broleo/go/bin:/usr/local/nodejs/node8/bin`
  - `SHELL`: `/bin/bash`
  - `LANG`: `en_US.UTF-8`
  - `PWD`: `/working/software/broleo/devel/system/listmycmds`


---------------------------

Contents of `Pipfile` ('/working/software/broleo/devel/system/listmycmds/Pipfile'):

```toml
[[source]]

verify_ssl = true
name = "pypi"
url = "https://pypi.python.org/simple"


[dev-packages]



[packages]

argh = ">=0.26.2"

```


Contents of `Pipfile.lock` ('/working/software/broleo/devel/system/listmycmds/Pipfile.lock'):

```json
{
    "_meta": {
        "hash": {
            "sha256": "c55ba166f3e7a75e9fef7b7f3212a61d66b6712edac5bd57ccc93c13f0668206"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "argh": {
            "hashes": [
                "sha256:a9b3aaa1904eeb78e32394cd46c6f37ac0fb4af6dc488daa58971bdc7d7fcaf3",
                "sha256:e9535b8c84dc9571a48999094fda7f33e63c3f1b74f3e5f3ac0105a58405bb65"
            ],
            "index": "pypi",
            "version": "==0.26.2"
        }
    },
    "develop": {}
}

```
</details>

[techalchemy]

Hi, sorry for the mix up. This isn’t actually a bug, but I’m just curious — can you provide some insight into why you thought run would also install packages? I don’t think this is likely to change but we are interested in improving the experience for new users, and we could improve messaging if this fails for example.

[halloleo]

I was reading somewhere in the doco that run makes sure all pacakges are installed. that's where my hope came from.

But if it is not there, that's ok. Pipenv is a great tool in any case! (However it would be a cool option for pipenv run, if it could make sure the venv has everything installed...)

[uranusjr]

Just realised this is basically #1924 but phrased differently… So it is not unreasonable to expect this intuitively, I guess. It would be a little tricky to get this right, however. It is probably not a good idea to always force the environment to stay up-to-date, and I fear the check could take a significant time. Given this overhead would be required for every run call, it needs to be as little as possible.

[uranusjr]

(Closing so there is a clear venue for discussion.)

[techalchemy]

Well it’s a discussion that we already had some time ago and decided not to auto install. The current behavior was a design decision. A warning could work

[halloleo]

What about an command-line option to pipenv run, say --force, which forces the virtual env to stay up-to-date?

[uranusjr]

@halloleo As a general rule run should not have any options because it could conflict with the option of the command you try to run.

[erinxocon]

I'm going to have to agree with @uranusjr on this one.

[techalchemy]

Also we have an entire segment of the api dedicated to installing things. I think it confuses people when the usage crosses otherwise clear api boundaries

[halloleo]

Yes, makes sense. After all I always can use a wrapper which calls pipenv update followed by pipenv run`.

[code-tree]

Can I suggest not creating the virtualenv at all then? Creating the env without installing packages can only be misleading as developers will assume packages were installed, and there is no way an app is going to run if its dependencies are not installed. Failing due to no env would be much clearer to debug.

[techalchemy]

For people who use npm or yarn, it’s not uncommon to put a build script in your package file. What if users ate currently using this to chain multiple steps, one of which is pipenv install? You don’t need all packages installed in order to run a script. I do appreciate it’s not necessarily intuitive to _everyone_ but I think it captures the majority use case and changing it would break too many things

[uranusjr]

I feel that is a bad way to use pipenv run though. I’d hope we can discourage people from doing this at all.

[techalchemy]

Agreed. I can’t remember why we landed on this as the behavior, and a cursory search wasn’t too helpful.

[code-tree]

@techalchemy So you mean they pipenv run python build_script.py with the script including pipenv install? Fair enough, I can see the benefits of that. Though it does still feel like catering to the needs of those users but not going further and catering to the "I need my packages installed" users :slightly_smiling_face:

[code-tree]

Would another option be having a post-install hook, where pipenv install is the first and only command you run, and Pipfile includes a post-install hook that can run build_script.py? Then the build script would have access to project dependencies (which would be very advantageous to many).

[uranusjr]

Wouldn’t this work just as well—

pipenv --three && pipenv run python build_script.py

I say we simply stop creating virtual environments for run and shell (and just error out). The alternative to the existing behaviour is straightforward and more explicit, and the change prevents a source of confusion.

[techalchemy]

For the record I originally agreed with you, but it was decided that this behavior is the expected behavior and that basically all of the interactions you have with pipenv should make sure you have a virtualenv (as far as these commands go)

[dbrgn]

As another voice in this issue, I also think that the current behavior (run will create a virtualenv but not install dependencies) is very confusing.

[JonasOlson]

So, what _is_ the intended way to run the program, and have dependencies installed automatically along the way? Is it pipenv install && pipenv run python myprog.py? It's a handful for being the thing that, in my view, i what you'd almost always want to run, the equivalent of make run or cargo run.

So, is there a shorter way?

Edit: Or maybe it is pipenv sync && pipenv run python myprog.py that I should run all the time.