pip unexpectedly not installing latest version of git package with branch/commit pinning

[dukebody]

(I'm posting this as a bug since nobody answered to http://stackoverflow.com/questions/30104748/pip-unexpectedly-not-installing-latest-version-of-git-package-with-branch-commit, so I think it is a bug, sorry if it isn't.)

I have a requirements.txt file with the following line (among others):

git+https://github.com/lead-ratings/sexmachine.git@master#egg=SexMachine

When I do

pip install -r requirements.txt

I see

Requirement already satisfied (use --upgrade to upgrade): SexMachine from git+https://github.com/lead-ratings/sexmachine.git@master#egg=SexMachine in /home/myuser/virtual_env/lib/python2.7/site-packages (from -r requirements.txt (line 38))

And the package is not updated to the master version. Actually, it keeps some former version from PyPI I had listed in requirements.txt before.

It doesn't work either if I specify a commit in the pinning or use the --no-cache-dir flag. I'm using pip 6.1.1.

If I use the --upgrade flag then it works. But then what is the point of the pinning? Why does it say "Requirement already satisfied" if it really isn't? I'd expect pip to always install the version specified in the pinning.

[ryneeverett]

I think this is a duplicate of #2835.

[dukebody]

Yes, it looks like so. Then I would advocate for documenting this behaviour in https://pip.pypa.io/en/latest/reference/pip_install.html#vcs-support, since it's an easy pitfall.

[ryneeverett]

+1

[ryneeverett]

@dukebody, what do you think of my proposed text?

[dukebody]

Hmmm it is not entirely clear IMO. What about the following?

*
For non-editable installs, the project is built locally in a temp dir and then installed normally. Please note that passing a specific target branch, tag or commit will only be used by pip to checkout that revision in the local temp dir and compare it against the currently installed version. If the same version number of the package (obtained from its setup.py) is already installed, the checked out VCS source will not overwrite it without an --upgrade flag.
*

[ryneeverett]

I think docs ought to explain in terms of behavior rather than implementation. My biased opinion is that my version is more clear.

[dukebody]

The sentence I think is not clear (at least to me) is "VCS requirements pin the version of the target commit, not necessarily the commit itself". What does "the version of the target commit" mean?

Perhaps we should invite someone else to this conversation to give some more input about how to explain this in clear language.

[dukebody]

BTW, if the latest code in the checked out source has a greater version number, will the new code be installed even if no --upgrade flag was specified? Since the requirement of the package being installed would be already met.

[ryneeverett]

The sentence I think is not clear (at least to me) is "VCS requirements pin the version of the target commit, not necessarily the commit itself". What does "the version of the target commit" mean?

Agreed. How about "the version (specified in the setup.py) of the target commit"?

BTW, if the latest code in the checked out source has a greater version number, will the new code be installed even if no --upgrade flag was specified? Since the requirement of the package being installed would be already met.

Yes, that is correct.

[dukebody]

Agreed. How about "the version (specified in the setup.py) of the target commit"?

Better. Can we settle this with the following?

*
Note that if a satisfactory version of the package is already installed, the VCS source will not overwrite it without an --upgrade flag. VCS requirements pin the package version (specified in the setup.py file) of the target commit, not necessarily the commit itself.
*

[ryneeverett]

@dukebody beautiful!

[skolsuper]

In my opinion, this is not a good resolution of this ticket. This behaviour breaks the rule of least astonishment, because it goes against specificity. If this were CSS, the version number in setup.py would be a .class, and the tag referenced in requirements.txt would be an #id. The version number is set by the maintainer and applies to everyone that uses the package, the tag in requirements.txt is set by the user and applies only to this environment. It is much more specific.

Can you give me an example scenario where a user might target a specific commit and actually want to just keep the package they already have?

I think the software should respect the specificity in the requirements.txt file. If that's not feasible, it should at least not fail silently and should alert the user that their requested branch/tag/commit was not respected.

FWIW, I struggled for 4 hours yesterday with a package and neither bumping the version number in my targetted branch, nor passing the -U flag worked. I still haven't got my preferred version installed even now.

Here is my fab script that sets up the environment:

def setup_venv(build):
    with cd('/webapps/{0}'.format(APP_NAME)):
        if exists('venv'):
            sudo('rm -R venv')
        sudo('virtualenv venv')
        for key, value in ENV_VARS[build.lower()].items():
            var_string = 'export {0}="{1}"'.format(key, value)
            append('venv/bin/activate', var_string, use_sudo=True)
        with prefix('source venv/bin/activate'):
            sudo('pip install -U pip')
            sudo('pip install -U -r requirements.txt -r prod-requirements.txt')

and here is the requirement I can't seem to install properly:

git+git://github.com/skolsuper/pybbm.git@tf-master#egg=pybb

I seem to end up with just the master branch every time. (In case you check the repo, I already tried bumping the version to 999, and rolled back that commit when it didn't work)

[ryneeverett]

@skolsuper I'm not sure why it wouldn't have worked when you bumped the version, but as a workaround I'd try making the requirement editable:

-e git+git://github.com/skolsuper/pybbm.git@tf-master#egg=pybb

As for resolving this ticket, how would you suggest pip keep track of what commit is currently installed in a noneditable package? Pip currently doesn't persist any state. Is this a good enough reason to start? Take a look at #453 and #507 in which the maintainer essentially closes his own PR to add state for this purpose.

My conclusion to this issue is to only use editable requirements. Editable requirements have the expected behavior (as you described it) and the additional advantage of pip freeze knowing the repository path. I'd like to see noneditable vcs requirements deprecated, but garnering support for feature removal is difficult.

[skolsuper]

Thanks @ryneeverett, I will give that a try. Is there a downside to using editable requirements, or is it a complete no-brainer? I've always avoided it on the basis that I won't be editing them so I don't need/want them editable.

[ryneeverett]

@skolsuper The only downside I'm aware of is that you have the entire git repository still on your system whereas with noneditable installs you just end up with the source. So it's not maximally lean but I'd say that cost is generally negligible. Hopefully we'll cut it down even further eventually with shallow clones.

[fgaudin]

So far I've always been using editable requirements to solve that issue but I just run into the downside of it:
we have https://github.com/django-guardian/django-guardian as a git requirement. When installing with -e, it will have that utils.py at the root which masks a utils package we have in our project.

If you set it as non editable, it only exports subpackages and everything is fine but then, changing the commit is not necessarily reflected.

[pztrick]

@dstufft I am raising this issue to your attention since you seem involved in the #2432 conversation regarding shallow Git clones as well as P/Rs (now closed) #453/#507 that are possibly relevant for a solution to this issue. Today I was burned by updating a Git checkout in requirements.txt like so:

# old:
git+git://github.com/celery/celery@d7d4c2a40ee4f482ae043f92e2a5391d395027a7#egg=celery[redis]
# new:
git+git://github.com/celery/celery@120770929f4a37c5373a378b75b5c41a99702af9#egg=celery[redis]

As discussed here, this will not re-fetch the new Git source code. The work-around stated above to pass the editable flag does work:

-e git+git://github.com/celery/celery@120770929f4a37c5373a378b75b5c41a99702af9#egg=celery[redis]

Of course... now I have a 54MB tree (.git directory) instead of a smaller 6MB tree (which will be duplicated in all my succeeding Docker images). I'd be happy to pass some other flag like -s git+git://... for a shallow-clone, but, failing that, perhaps PR #507 makes sense to know if source should be re-fetched (to resolve this issue)?

[jaraco]

Today I encountered another condition where an editable install is inadequate - when environment markers are present. Consider this requirement:

git+https://github.com/cannatag/ldap3@dev#egg=ldap3;python_version=="2.7"

This requirement will not update if some version of ldap3 is already in the environment. Prepending -e to make the requirement editable results in InvalidRequirement: Invalid requirement, parse error at "';python_'".

[BrunoDesthuilliers]

Ok, I (more or less) understand why pip works that way, but I have to say it _really_ breaks expectations in a major way and that the proposed solution (using editable installs) is a workaround at best, with obviously quite a few downsides, cf the previous comments. Not such a problem in my case ATM (the few packages I use with commits as version are small) but if someone can think of possible solution to really fix this issue I'll be happy to give a hand.

[carsonip]

BTW, if the latest code in the checked out source has a greater version number, will the new code be installed even if no --upgrade flag was specified? Since the requirement of the package being installed would be already met.

Yes, that is correct.

Even if the checked out source has a greater version number, the code will not be installed unless there is the --upgrade flag. Actually it seems that even when the version number didn't change, it is still installed.

[mitar]

I can confirm that package does not really upgrade even if version in setup.py is different from what is already on the system. Using pip 19.0.3. So even using version inf #egg which is different from one used before does not help.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.