Pipenv: Pipenv unexpectedly installs from requirements.txt (even from parent directories)

[askpatrickw]

This was a weird one and it was so odd that I thought I was losing my mind for a bit. ;-) Thanks for pipenv though, I'm very excited about this new direction!

Pipenv installs requirements.txt when you don't ask it to, and when it try to keep it from installing them, it still does. This is on Windows 10. Not sure if reproduces on LinuxOSx

$ python -m pipenv.help

$ python -m pipenv.help output

Pipenv version: '2018.05.18'

Pipenv location: 'C:\\Users\\patri\\AppData\\Local\\Programs\\Python\\Python36\\lib\\site-packages\\pipenv'

Python location: 'C:\\Users\\patri\\AppData\\Local\\Programs\\Python\\Python36\\python.exe'

Other Python installations in PATH:

• 3.6: C:\Users\patri\AppData\Local\Programs\Python\Python36\python.exe

• 3.6.5: C:\Users\patri\AppData\Local\Programs\Python\Python36\python.exe

• 3.6.5: C:\Windows\py.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.17134',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'win32'}

System environment variables:

• ALLUSERSPROFILE
• APPDATA
• COMMONPROGRAMFILES
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• COMPUTERNAME
• COMSPEC
• DRIVERDATA
• GDAL_DATA
• HOMEDRIVE
• HOMEPATH
• LOCALAPPDATA
• LOGONSERVER
• NUMBER_OF_PROCESSORS
• ONEDRIVE
• OS
• OSGEO4W_ROOT
• PATH
• PATHEXT
• POSTGIS_ENABLE_OUTDB_RASTERS
• POSTGIS_GDAL_ENABLED_DRIVERS
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMDATA
• PROGRAMFILES
• PROGRAMFILES(X86)
• PROGRAMW6432
• PROJ_LIB
• PROMPT
• PSMODULEPATH
• PUBLIC
• SESSIONNAME
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TMP
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• VS140COMNTOOLS
• WINDIR
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\iCLS\;C:\Program Files\Intel\Intel(R) Management Engine Components\iCLS\;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Windows Resource Kits\Tools\;C:\ProgramData\Oracle\Java\javapath;C:\Program Files\Dell\DW WLAN Card;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\WIDCOMM\Bluetooth Software\;C:\Program Files\WIDCOMM\Bluetooth Software\syswow64;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Users\patri\Tools\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\Program Files (x86)\Microsoft VS Code\bin;C:\Program Files\Amazon\AWSCLI;C:\Users\patri\AppData\Local\GitHub\PortableGit_d76a6a98c9315931ec4927243517bc09e9b731a0\cmd;C:\Users\patri\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Nmap;C:\Users\patri\AppData\Roaming\npm;C:\Program Files\Heroku\bin;C:\OSGeo4W64\bin;c:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\;c:\Program Files\Microsoft SQL Server\100\Tools\Binn\;c:\Program Files\Microsoft SQL Server\100\DTS\Binn\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Amazon\AWSCLI\bin\;C:\WINDOWS\System32\OpenSSH\;c:\users\patri\appdata\local\programs\python\python36\Scripts;C:\Users\patri\AppData\Local\Programs\Python\Python36\Scripts\;C:\Users\patri\AppData\Local\Programs\Python\Python36\;C:\Program Files\Amazon\AWSCLI;C:\Users\patri\AppData\Local\GitHub\PortableGit_d76a6a98c9315931ec4927243517bc09e9b731a0\cmd;C:\Users\patri\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Nmap;C:\Program Files\Heroku\bin;C:\Users\patri\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Microsoft VS Code\bin;C:\Users\patri\AppData\Local\Pandoc\;

---

Contents of Pipfile ('C:\Users\patri\code\local\piptest\Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
nflgame = "==1.2.20"
requests = "*"

[dev-packages]

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('C:\Users\patri\code\local\piptest\Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "dd4fd0a58e50c39e99da316cc62783d7f58d2a7cfd3fdf5965233e3fcef3f88d"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "beautifulsoup4": {
            "hashes": [
                "sha256:11a9a27b7d3bddc6d86f59fb76afb70e921a25ac2d6cc55b40d072bd68435a76",
                "sha256:7015e76bf32f1f574636c4288399a6de66ce08fb7b2457f628a8d70c0fbabb11",
                "sha256:808b6ac932dccb0a4126558f7dfdcf41710dd44a4ef497a0bb59a77f9f078e89"
            ],
            "version": "==4.6.0"
        },
        "certifi": {
            "hashes": [
                "sha256:13e698f54293db9f89122b0581843a782ad0934a4fe0172d2a980ba77fc61bb7",
                "sha256:9fa520c1bacfb634fa7af20a76bcbd3d5fb390481724c597da32c719a7dca4b0"
            ],
            "version": "==2018.4.16"
        },
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "version": "==3.0.4"
        },
        "httplib2": {
            "hashes": [
                "sha256:e71daed9a0e6373642db61166fa70beecc9bf04383477f84671348c02a04cbdf"
            ],
            "version": "==0.11.3"
        },
        "idna": {
            "hashes": [
                "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f",
                "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4"
            ],
            "version": "==2.6"
        },
        "nflgame": {
            "hashes": [
                "sha256:0a9852009f71870ac3e44e9aeca4f4664c76accd2353fc7c4c9dd42f83e163a7",
                "sha256:1015675386dd97341f02e1614ef79f8b162cb2826bbddc8c2fec3de0eff20055"
            ],
            "index": "pypi",
            "version": "==1.2.20"
        },
        "pytz": {
            "hashes": [
                "sha256:65ae0c8101309c45772196b21b74c46b2e5d11b6275c45d251b150d5da334555",
                "sha256:c06425302f2cf668f1bba7a0a03f3c1d34d4ebeef2c72003da308b3947c7f749"
            ],
            "version": "==2018.4"
        },
        "requests": {
            "hashes": [
                "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
                "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e"
            ],
            "index": "pypi",
            "version": "==2.18.4"
        },
        "urllib3": {
            "hashes": [
                "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
                "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
            ],
            "version": "==1.22"
        }
    },
    "develop": {}
}

Expected result

pipenv install should NOT install anything else other than library & dependencies.
pipenv install should not install requirements.txt when requirements.txt has been moved (from the repro and not commited)

Actual result

pipenv installs requirements.txt when it exists and its not asked to.
pipenv installs requirement.txt when it is deleted and the change is not commited to the repro

When possible, provide the verbose output (--verbose), especially for locking and dependencies resolving issues.

Steps to replicate

1. Git init new_repro
2. Create requirements.txt and add a library (nflgame==1.2.20 for example)
3. Don't pip install or pipenv install anything at this point.
   4.. Add and Commit requirements.txt to the repro
4. move requirements.txt out of the repro.
   .6 pipenv install some other library (pipenv install requests)

The library in the requirements.txt is installed
```
pipenv graph
nflgame==1.2.20

• beautifulsoup4 [required: Any, installed: 4.6.0]
• httplib2 [required: Any, installed: 0.11.3]
• pytz [required: Any, installed: 2018.4]
  requests==2.18.4
• certifi [required: >=2017.4.17, installed: 2018.4.16]
• chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
• idna [required: <2.7,>=2.5, installed: 2.6]
• urllib3 [required: <1.23,>=1.21.1, installed: 1.22]
  ``

[uranusjr]

I can reproduce on macOS. This is actually kind of cool, but we still need to fix it.

[uranusjr]

I changed the title a little to reflect the situation better.

[uranusjr]

I think I know what’s going on here. The requirements.txt finder looks in parent directories (not sure how many levels it would look) for requirements.txt. I’m not sure if this is the intended behaviour, but at least this seems unrelated to Git. Modifying the title again to reflect this.

[askpatrickw]

We may want to split this out into an additional bug

I also noted that when I pipenv install I don't want it to find requirements.txt and install everything. The reason I found this issue to begin with was I wanted to _not_ install all the things in my requirements.txt because I wanted to install things manually so I could manage what was in Pipfile vs. the dependencies and wanted to move other things to --dev.

I probably should have put that in the bug description to begin with.

[techalchemy]

Pipenv recurses up to find pipfiles and requirements files. You can set PIPENV_MAX_DEPTH but if you have a Pipfile it should ignore requirements files altogether unless you specify one

[askpatrickw]

Hi @techalchemy, that's a nice explanation of how things work now.

I've tried pipenv several times over the past year and kept switching back to pip and virtualenvwrapper-win because of these behaviors I didn't understand. I think it would be improved, especially for new Python developers if the tool was more deterministic and if the actions it performed were explicit rather than implied.

Some ideas:

• Maybe pipenv should not search parent folders at all. I don't even understand how this could be handled reliably. What if I had a nested set of repros (like when you do git clone --recursive)? And why does it only search up? Maybe this would be better as optional flags --search-direction up|down --search-depth int
• I definitely think, Pipenv should not _assume_ I want install something when I explicitly state things I do want to install. So don't install requirements.txt (without asking me) if I explicitly state to install something. Maybe pipenv should have a explicit --requirements flag (so I can install X and --requirements) to do that though.

Thanks again for pipenv!

[uranusjr]

I feel the “have a requirements.txt but not install everything” feature is an edge case. You can create the Pipfile manually in this case (touch Pipfile if you are on POSIX) to avoid the detection, or if #2188 is accepted, use the init command with the same effect.

Regarding to the auto detection, however, I do find the current “search in parent directories” behaviour counter-intuitive. The auto-import only happens when you first initialise the environment (i.e. don’t have a Pipfile yet), and in almost all situations you would do it in the project root, because that just makes sense.

How Pipenv converts requirements.txt in a parent directory is also not very consistent with itself. Say I have this file structure:

parent
├── project
└── requirements.txt

When I run pipenv install in project, parent/requirements.txt will be imported, but the resulting Pipfile would be in project. If we search requirements.txt in parent directories, we ought to consider where we find it the project root, and create the Pipfile in that directory instead. If initialisation always create Pipfile in the current directory, then we need to treat the directory where we initialise as the project root, and don’t search outwards for requirements.txt.

So there are two ways to fix this inconsistency (that I can think of):

• Initialisation should create Pipfile in the same directory where if finds a requirements.txt, if there is one.
• Initialisation should only consider requirements.txt in the current directory, not in parent directories. The result Pipfile will be created where you run the initialisation command (the current behaviour).

[techalchemy]

I agree that we can reevaluate most of this. It definitely causes confusion

[askpatrickw]

I spent a little time messing around and wanted to add some notes...

I see now that when you pipenv install library in a new directory it ALWAYS creates Pipfile and Pipfile.lock, So, if you then pipenv --rm and pipenv install new_library it will then always install the new_library and everything in the Pipfile.

Coming from the pip & virtualenv world that wasn't obvious to me. Based on a deeper understanding, I _sorta_ get why if there was no Pipfile and pipenv finds a requirements.txt it wants to treat it like a Pipfile and process it.

Based on an earlier comment, I thought maybe I could workaround by using a blank file. I created a blank Pipfile with a requirements.txt in the same folder and the libraries in requirements.txt where installed when I pipenv installed a new library.

I go back to my original use case and I think I'm at the same opinion:

• I have a project with a requirements.txt what was created using pip freeze and I want to move to pipenv.
• I want to manually install my libraries with pipenv so I can end up with a proper Pipfile that only has my requirements and not my requirements dependencies. Because when the requirements.txt is processed you end up with Pipfile with all the dependencies added.

Originally, I moved the requirements.txt up a folder and we see that didn't work. As-is I guess you have to rename requirements.txt or delete it.

Instead, I think pipenv should never automatically process a requirements.txt and instead an --import-requirements (or something) should be added to be used in the migration process.

However, maybe there is another use case I'm not considering, like trying to co-exist both Pipfiles and Requirements.txt?

Thanks for the good discussion!

[uranusjr]

While I understand it can be technically cleaner to have an explicit --import-requirements option, I can see how auto-importing is good for adoption. Importing requirements.txt is the right thing nine times out of ten, and it creates a nice wow effect. It could be a good idea to modify this in the future (after Pipfile is at least as adopted as requirements.txt), but for now it is probably best to keep the current behaviour IMO. It is probably better to make this behaviour a bit more smart, i.e. import only when it is really sure the requirements.txt is in the project.

[techalchemy]

We have an internal toggle to ignore requirements files. We probably should ignore them when a user passes in something to install

[uranusjr]

That sounds reasonable.

[carlosdanielcsantos]

This behavior is extremely error-prone. I was chasing ghosts for hours until I realized that a requirements.txt was laying around in a parent folder that didn't have anything to do with the project I working on...

pipenv, version 2018.7.1

Unfortunately, this was not fixed by #2309 . Calling pipenv install in a directory that has a Pipfile still installs from parent's requirements.txt

[uranusjr]

Feel free to submit a PEEP about this and suggest an alternative.

[martinraag]

I have to voice my support for @carlosdanielcsantos opinion here. Pipenv looking for requirements.txt file in parent directories is extremely counter intuitive for me. I just ran into this problem as well.

In my opinion it would be reasonable to treat the current working directory as the project root when running pipenv and not look for requirements.txt, or anything else for that matter, in parent directories.

[uranusjr]

Feel free to submit a PEEP about this and suggest an alternative.

[martinraag]

@carlosdanielcsantos are you working on a PEEP for this already?

[carlosdanielcsantos]

Not yet @martinraag, please go ahead!

[washeck]

I was also hit by this. I understand that changing pipenv behaviour requires some considerations and effort but I feel it would help a lot if pipenv at least print the absolute path to the found requirements.txt

The message _requirements.txt found, instead of Pipfile! Converting…_ is extremely confusing when there is no requirements.txt in current folder.

[uranusjr]

print the absolute path to the found requirements.txt

That’s likely a good idea. Care to submit a PR? I imagine it’d be appropriate to output something like:

requirements.txt found! (in <directory>)
Coverting to Pipfile…