pip 8.1.2 breaks project because of no backward compatibility in package resource API

[viyatb]

• Pip version: 8.1.2
• Python version: 2.7.11
• Operating System: Kali Linux
  

  Description:

Well, I upgraded to the latest pip version and our project won't run because of breaking changes to the API - specifically to the Requirements object. The code was something like this -

def verify_dependencies(root_dir):
    # Get all the installed libraries
    # installed_libraries = {"tornado": "version"}
    installed_libraries = dict(
        (i.project_name, i.version) for i in pip.get_installed_distributions())

    # Get all the libraries required by owtf
    # owtf_libraries = ["tornado", "lxml"...]
    owtf_reqs = pip.req.parse_requirements(
        os.path.join(root_dir, "install", "owtf.pip"),
        session=uuid.uuid1())
    owtf_libraries = [req.req.project_name for req in owtf_reqs]

    # Iterate over requirements and check if existed
    missing_libraries = []
    for library_name in owtf_libraries:
        if library_name not in installed_libraries.keys():
            # Check if the module is installed via package manager
            if not is_present(library_name):
                missing_libraries.append(library_name)

    # If there are missing libraries bail out :P
    if len(missing_libraries) > 0:
        print("The following python libraries seem missing : ")
        print("   %s\n" % (','.join(missing_libraries)))
        print("Haven't you run the install script? ")
        print("   %s\n" % ("python2 install/install.py"))
        print("If you are sure you ran the install script, "
              "install the missing libraries seperately")
        print("   %s\n" % ("pip install --upgrade -r install/owtf.pip"))
        exit(1)

Now the owtf_libraries object is a list of elements (req here) of a generator which are InstallRequirement objects. The req is a Requirements object.

The error was:

owtf_libraries = [req.req.project_name for req in owtf_reqs]
AttributeError: 'Requirement' object has no attribute 'project_name

I have fixed the error by using req.req.name instead of req.req.project_name .But please maintain backward compatibility so that projects do not break with any changes pip has. In the past, almost every time pip had a breaking change, projects break.

[RonnyPfannschmidt]

pip officially has no stable python api

this is a duplicate of the issue the saltstack guys reporte a few days back

the proposed solution is to write an abstraction layer

[dstufft]

Closing as their is no internally stable API for pip.

[viyatb]

@dstufft @RonnyPfannschmidt I concur with @kiorky. The point is pip is a packaging software, so almost all of open-source and private libraries depend on this. I have seen libraries, tools breaking because of pip just days after their stable release (happened twice in my case with owtf ). In any case, pip should keep the breaking changes to major version changes and also provide a deprecation warning before changing the internal API however they want.

[dstufft]

The entire point of the distinction of an internal API (versus a public API) is an internal API is not backed by any promise of stability and can change at any time, even among the smallest of patch releases. If there is an internal API that a project would like to depend on, the right answer is to propose making it a public API (or making a new API that exposes the same functionality, and is public). The wrong answer is to expect maintainers to maintain compatibility for an API that they've not said they will just because you happened to use that API.

[dstufft]

Breaking it is just retard.

Please do not use "retarded" as a pejorative. You're expected to follow the PyPA Code of Conduct when participating in a PyPA owned space.

Thinking to have Requirement patch class that bring retrocompat compatiblity is not hard to maintain but just make your software not breaking everything out there.

The problem is that within a definition of what the public API is, it's impossible to actually make any changes what so ever, because anyone could be relying on any random implementation detail and expect it to not change. The public/internal distinction exists to it possible to actually balance between making no changes ever and being able to refactor and adjust the code to enable new things.

I expect from a packaging lib to not annoy me because of regressions, which pip does a lot, and the worse in there is that it is wanted.

If you don't feel pip is living up to the standard you wish it to, you're free to use another tool or to fork it and maintain it yourself.

[viyatb]

@dstufft So should I open an issue for a standard public API for pip or is this already in works?

[dstufft]

@delta24 If there's an API you'd like to be made public, yes open an issue with what API that is and what the use case is. That will let us evaluate if it's an API we want to make public, or if there's a better way to solve the same problem with a new, cleaner API, or the like.

[dstufft]

Our public API is generally the CLI for a few reasons:

• Pip modifies the state of Python, and that state change doesn't get picked up inside of an already running Python process. So once you've modified the state of things, you need to start a new process to get it picked up. This makes a programatic interface that isn't subprocess based difficult to use correctly.
• When people have asked for a public API for something in pip, it was generally for something that wasn't pip specific but some thing they wanted to do with packaging (one example is parsing versions) so instead we would spin that out into it's own library with a good, clean API and less gotchas and then consume that new library from pip itself.
• A lot of the code in pip is badly factored and isn't well suited to external use.

[pfmoore]

@kiorky The key point is that there is _nowhere_ in the documentation of pip that states that it is allowed to do "import pip" and use functions exposed by that import in your own program. As far as the documentation is concerned, pip is purely a command line tool - we could choose to rewrite it in C if we wished without violating any guarantee made by the documentation.

If you want us to provide documentation stating the functions that should be usable from 3rd party code, that's fine. We don't promise to _agree_ with your suggestions, but we will consider them. Alternatively, libraries such as packaging _are_ designed to be used as libraries and have a documented API - one of those libraries may well suit your needs already.

But I don't find your tone appropriate given that you're using pip in a way that we gave you no promises about (in fact, it's been well established on many occasions that pip has no public API and we don't guarantee we won't change the internals at any time). Comments like "I expect from a packaging lib to not annoy me" (to say nothing of your even less appropriate language) is not likely to get much sympathy. Your repeated complaints that we "broke" code are completely incorrect. The affected users were using pip in a way that we don't support, have never said we support, and indeed we've repeatedly _said_ we don't support - _they_ took the risk, and therefore they need to accept the consequences. Most other people affected have done so politely, and worked with us to find a solution. You're the exception.

For what it's worth there's history here that you might not be aware of. Distutils allowe users to work with its internal API, in spite of the fact that it didn't document the public interface. As a result, people used virtually every part of distutils, making it impossible to change anything without breaking code (and in distutils' case, the authors _had_ committed to not doing that). The result is an unmaintainable mess. Maybe we're being excessively cautious, but one reason we don't allow unrestricted use of the pip internals is because we don't want to fall into that trap (and the reason we don't document a clear, carefully designed API that we _will_ support is a combination of lack of time, and lack of good feedback from people who might use such an API).

[viyatb]

@pfmoore @dstufft I understand your position but I think that at least the breaking change detail should be announced in the release notes (Github releases) so that we know how and where to fix the error without going into the pip internals. Is this reasonable? :)

[pfmoore]

If by "the breaking change" you mean the change to which Requirements class we used, I see your point, to an extent, but how do we know what breaking changes count as worth documenting, given that nobody should actually be using _any_ of this interface? Which of the many other internal changes might break someone's code? How would we know?

I can only repeat my earlier comment - if people want to propose which APIs we should be documenting and/or making supported (presumably because they use them in their code) we can have a discussion. But vague requests that we "have a supported API" won't go anywhere, because we don't know what the user requirements are.

[HolgerPeters]

@delta24 I am writing this because you seem to insist that this is a breaking change. While I understand that suffering from a change in private API can be frustrating, the key point here is that a change in the internals is not a breaking change of the public API. Pip's public API is the command line interface, so there are no guarantees about classes and functions for pip. Changes that are not affecting a user of the public API should not be communicated in release notes or reflected in the versioning scheme. Otherwise, the distinction of private and public API would lose any meaning. And it is very important to allow maintainers to change the internals, because otherwise maintainers couldn't make even simple changes in a code base.

If you want to read up on this, you might look into https://www.python.org/dev/peps/pep-0008/#public-and-internal-interfaces and semver.org. Regarding you request, PEP8 directly states:

All undocumented interfaces should be assumed to be internal.
Which is the case with pip's classes.

[viyatb]

@dstufft @pfmoore @HolgerPeters sorry for the confusion. I know that while the changes in the internal classes and functions are of no use to the end user, but it would super useful for someone following pip's internal development ie. the developers.
Other than this, your arguments are valid.

[RonnyPfannschmidt]

@delta24 you might want to consider a CI setup that automatically tests your usage against each "green" commit of pip - unlike manually maintained release notes (which need to be both written exthausively and read exthuasively) that cannot be forgotten