pipenv lock --keep-outdated updates subdependencies

[MartinAltmayer]

Issue description

pipenv lock --keep-outdated updates versions of subdependencies despite the --keep-outdated flag.

Example: Pipfile contains python-dateutils which has six as a dependency. If a newer version of six is available it will be put into Pipfile.lock by pipenv lock --keep-outdated.

Expected result

When used with --keep-outdated, pipenv lock uses the currently installed versions without doing any updates.

Actual result

Pipenv lock puts newer versions of subdependencies into Pipfile.lock.

Steps to replicate

• Put this Pipfile into an empty folder

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
python-dateutil = "==2.7"
six = "==1.5"

[requires]
python_version = "3.7"

• pipenv install
• Pipfile.lock will contain `six = '==1.5'
• delete the line `six = '==1.5' from the Pipfile (this simulates a release of a newer version of a subdependency)
• pipenv lock --keep-outdated
• Pipfile.lock will now contain six = '==1.12'.

This issue seems to apply only to subdependencsies: If you specify "*" for python-dateutil in the Pipfile, pipenv lock --keep-outdated will (correctly) refrain from updating python-dateutil.

---

$ pipenv --support

Pipenv version: '2018.11.15.dev0'

Pipenv location: '/usr/lib/python3.7/site-packages/pipenv'

Python location: '/usr/bin/python'

Python installations found:

• 3.7.4: /usr/bin/python3
• 3.7.4: /usr/bin/python3.7m

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_system': 'Linux',
 'python_full_version': '3.7.4',
 'python_version': '3.7',
 'sys_platform': 'linux'}

System environment variables:

Pipenv–specific environment variables:

Debug–specific environment variables:

---

Contents of Pipfile ('/home/martin/temp/piptest/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
python-dateutil = "==2.7"

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('/home/martin/temp/piptest/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "458ffc4c79d86d5ffe6888bcc057e0dfdf5ef89c0728cd0ea600dfebb33042dd"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "python-dateutil": {
            "hashes": [
                "sha256:07009062406cffd554a9b4135cd2ff167c9bf6b7aac61fe946c93e69fad1bbd8",
                "sha256:8f95bb7e6edbb2456a51a1fb58c8dca942024b4f5844cae62c90aa88afe6e300"
            ],
            "index": "pypi",
            "version": "==2.7"
        },
        "six": {
            "hashes": [
                "sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
                "sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
            ],
            "version": "==1.12.0"
        }
    },
    "develop": {}
}

[DusanMadar]

I can confirm the same behavior.

Tested a couple of scenarios

1. cp Pipfile.lock Pipfile.lock.original && pipenv lock --keep-outdated
   
   
   
   • Pipfile.lock different from Pipfile.lock.original
   
   
2. mv Pipfile.lock Pipfile.lock.keep_deps && pipenv lock
   
   
   
   • Pipfile.lock different from Pipfile.lock.original
   
   
   • Pipfile.lock different from Pipfile.lock.keep_deps
   
   

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/usr/local/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

• 3.7.5: /usr/local/bin/python
• 3.7.5: /usr/local/bin/python3.7m

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.0.0-36-generic',
 'platform_system': 'Linux',
 'platform_version': '#39~18.04.1-Ubuntu SMP Tue Nov 12 11:09:50 UTC 2019',
 'python_full_version': '3.7.5',
 'python_version': '3.7',
 'sys_platform': 'linux'}

The OS is actually Alpine Linux 3.10.3 (Dockerfile FROM python:3.7-alpine)

[mcallaghan-bsm]

Can confirm this as well. It is super undesired. I think the key issue occurs when application/default packages share sub-dependencies with dev/develop packages.

The only WORKAROUND I found was to manually revert the "default" JSON section (short of figuring out some complex bash/git-patch script).

[jacobisaliveandwell]

@frostming Could this get some attention?

[bendem]

The problem makes sense, the goal of the lock is to update dependencies, the goal of pipenv lock --keep-outdated -d -r > requirements.txt is the opposite of that, we basically bastardized that command to do the opposite of what it was written for. Shouldn't that feature be moved to the sync command instead? sync reads the lockfile and installs dependencies, it could get a -r that writes a requirements file instead of installing.

[tilgovi]

The goal of lock is to generate a Pipfile.lock so that the current environment (working, hopefully) can be reproduced later. Ideally, this would generate a lock file that reflects the current state of installed packages by looking at what is actually installed right now. Nothing should be installed or upgraded unless it's currently missing entirely and therefore needs to be resolved.

The workflow that is still challenging is going from a Pipfile with semver ranges to a Pipfile.lock that pins the particular versions installed now, and it should be possible to selectively upgrade some dependencies and transitive dependencies and to cement those changes in the lock file without having other things updated unexpectedly.

[frostming]

Repeating the exact same steps on 2020.11.15, the issue has gone.

I am going to close this one, other comments are different issues, please track in #4539 . Thanks

[bendem]

The goal of lock is to generate a Pipfile.lock

Exactly my point, this feature requested is something that, if it updates the lockfile, is meaningless. The goal of the lock command is to resolve versions, this feature request is to never ever try to resolve versions but instead use the lockfile with all the versions already resolved.

[tilgovi]

The goal of the lock command is to resolve versions

I think this is the point of disagreement. The only time I expect it to resolve a version is when there is no existing lock file or the existing lock file entry does not satisfy an updated entry in the Pipfile. The request for this issue was to make it possible to make the minimal updates to satisfy the Pipfile, while keeping everything that already does satisfy unchanged.

[mcallaghan-bsm]

uhhhh

preconditions

sudo apt install python3.7-minimal
pip3 install pipenv

priming initial conditions

~/projects/pipenv_3975$ python3 -m pipenv install

$ cat Pipfile.lock 
{
    "_meta": {
        "hash": {
            "sha256": "3c64c9c09d36cfc9046ec8703b8bb5b757f7196e39c597ead4fb4e749b8b0332"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "python-dateutil": {
            "hashes": [
                "sha256:07009062406cffd554a9b4135cd2ff167c9bf6b7aac61fe946c93e69fad1bbd8",
                "sha256:8f95bb7e6edbb2456a51a1fb58c8dca942024b4f5844cae62c90aa88afe6e300"
            ],
            "index": "pypi",
            "version": "==2.7"
        },
        "six": {
            "hashes": [
                "sha256:7ba77770fedd5b84d6c67283912cad39856b5532dc1beb76182ccacee9ec41e0",
                "sha256:9fa05de9aa4e0042767dd666ce4905a097306b6291148e958162c4ca0b52d16d"
            ],
            "index": "pypi",
            "version": "==1.5"
        }
    },
    "develop": {}
}

then conduct the test, REMOVE six==1.5 from Pipfile
, and then try:

$ python3 -m pipenv lock --keep-outdated
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

Traceback (most recent call last):
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 760, in main
    dev=parsed.dev)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 712, in resolve_packages
    results = clean_outdated(results, resolver, project, dev)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 606, in clean_outdated
    reverse_deps = project.environment.reverse_dependencies()
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/project.py", line 376, in environment
    self._environment = self.get_environment(allow_global=allow_global)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/project.py", line 366, in get_environment
    environment.extend_dists(pipenv_dist)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/environment.py", line 127, in extend_dists
    extras = self.resolve_dist(dist, self.base_working_set)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/environment.py", line 122, in resolve_dist
    deps |= cls.resolve_dist(dist, working_set)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/environment.py", line 121, in resolve_dist
    dist = working_set.find(req)
  File "/home/USER_SNIPPED/.local/share/virtualenvs/pipenv_3975-PaIxvLxT/lib/python3.7/site-packages/pkg_resources/__init__.py", line 627, in find
    raise VersionConflict(dist, req)
pkg_resources.VersionConflict: (six 1.5.0 (/home/USER_SNIPPED/.local/share/virtualenvs/pipenv_3975-PaIxvLxT/lib/python3.7/site-packages), Requirement.parse('six<2,>=1.9.0'))

[jacobisaliveandwell]

@mcallaghan-bsm Your initial command is to install 3.7, but your trace is using 3.6. I think PEBKAC in this case.

[mcallaghan-bsm]

(kk abandoning issue closed NWA)