Pip: Temporarily restore --build-dir for PyCharm

[sfelixkim]

Environment

• pip version: 20.3
• Python version: 3.8
• OS: Windows 10

Description
Cannot install any packages when using PyCharm python interpreter.
The virtual environment is created using virtualenv.
No problem when executing pip install [package] on either PyCharm terminal or cmd prompt directly.
Problem resolved when pip is downgraded to 20.2 (had to be done on terminal by pip install pip==20.2.4)

Expected behavior
Install packages successfully.

How to Reproduce

1. create a new project on PyCharm (then the installed pip version is 20.3)
2. install any package (e.g. flask) by typing the name and click the "Install Package" button
3. An error occurs.

Output
executed command:

pip install Flask

Error occurred:

Non-zero exit code (2)

proposed solution:

Try to run this command from the system terminal. Make sure that you use the correct version of 'pip' installed for your Python interpreter located at 'C:\path\to\project\folder\venv\Scripts\python.exe'.

command output:

Usage:   
  C:\path\to\project\folder\venv\Scripts\python.exe -m pip install [options] <requirement specifier> [package-index-options] ...
  C:\path\to\project\folder\venv\Scripts\python.exe -m pip install [options] -r <requirements file> [package-index-options] ...
  C:\path\to\project\folder\venv\Scripts\python.exe -m pip install [options] [-e] <vcs project url> ...
  C:\path\to\project\folder\venv\Scripts\python.exe -m pip install [options] [-e] <local project path> ...
  C:\path\to\project\folder\venv\Scripts\python.exe -m pip install [options] <archive url/path> ...

no such option: --build-dir

[uranusjr]

See: https://mail.python.org/archives/list/[email protected]/message/6Z5DQGRDGOU4OCO3JSS4FVYWV5JBUPVP/

@pradyunsg Would it be a good idea to restore the option before PyCharm can release a fix on their end?

[gaborbernat]

Do we know why PyCharm developers did not see the warning about that option being deprecated? Maybe we should improve on that.

[jerryvurbl]

so this is basically a pycharm issue? Is 6 months warning enough? Probably. Should this critical breaking change be coordinated between the biggest IDE's developers and the developers of the most important executable in the python ecosystem? Probably. Where do we go from here? We have to open a PyCharm issue?

[gaborbernat]

Should this critical breaking change be coordinated between the biggest IDE's developers and the developers of the most important executable in the python ecosystem? Probably.

All major IDE developers have access to the pip changelog. The pip developer team has no way to know what deprecated interfaces these IDEs use (against emitting warnings for 6+ months), especially sometimes when those sources are behind closed doors. Yes, you should be opening a PyCharm issue. The issue lies with them at the end of the day. Should they perhaps have a test suite against the master branch/RC versions of pip? Probably.

[sbidoul]

Would it help if the option was still present as a no-op ?

[gaborbernat]

The PyCharm issue https://youtrack.jetbrains.com/issue/PY-45712. Seems there was a commit to remove the deprecated flag yesterday on the PyCharm code base https://github.com/JetBrains/intellij-community/commit/a618daac6a675202ca38c6b3ea343b70832cbc93

[pfmoore]

@sbidoul I was going to say yes, leaving the option as a no-op for a short while might be helpful. But if JetBrains have already committed the change, it sounds like they are in a position to release a fix, so I think we can leave it to them.

[pradyunsg]

Should this critical breaking change be coordinated between the biggest IDE's developers and the developers of the most important executable in the python ecosystem?

Ok, this annoys me more than I should probably allow it to.

pip's maintainers have communicated about this change across multiple public channels, (as noted earlier) this option has been printing warnings on every invocation for the past few releases and we even had a beta release for 20.3 out for nearly 1 month, where this option was removed. I'm genuinely not sure what more we could've done to avoid this situation from our end. From my perspective, no reports/requests came in related to this deprecation from end users, so this was supposed to be "the easy one" -- I'd even dropped it from our 20.3 release announcement publicity push, because I thought: well, this removal seemed to affect no one so why call it out in the emails (LOL). To imply that pip's devs didn't do enough feels exceedingly dismissive of our efforts, and is the kind of thing that results in folks walking away from projects.

---

sigh Sorry, I got worked up there.

---

@pradyunsg Would it be a good idea to restore the option before PyCharm can release a fix on their end?

Yes.

It seems like PyCharm has been able to make the changes on their end to accomodate for the removal, but I'd imagine it might still be necessary to reduce breakage as they roll out the fix. If someone from the PyCharm team could clarify/provide an answer to the question:

Would it help if the option was still present as a no-op ?

I don't want to reintroduce the --build-dir _functionality_ to be very honest -- that removal was blocking internal cleanups I'd really like to get to -- but maintaining a no-op option for a few releases is very "cheap" in terms of maintenance effort. Looking at the removal patch, it seems like this would be OK?

I'm happy to cut a pip 20.3.1 later today (assuming someone can file a PR adding this, with a comment pointing back to this issue), if that would help with this breakage for PyCharm's users.

[pfmoore]

To imply that pip's devs didn't do enough feels exceedingly dismissive of our efforts, and is the kind of thing that results in folks walking away from projects.

Agreed.

Can I also note that the comments I saw from the PyCharm developers were polite, reasonable, and not at all confrontational, so I'd like to publicly thank them for taking a completely reasonable approach here.

Drive-by criticism of either project over this from 3rd-party bystanders is neither helpful nor welcome, in my view.

[jerryvurbl]

Fair enough, will do.

That said maybe this should have been a semver major release change on the pip side?

it should be noted that a user has to adopt this pip change to break things and it can be rolled back in the same way.

maybe its me but i dont read all the release notes for the hundreds of projects i use and track but a major release increment usually gets for attention. a breaking change appearing between x.2.4 to x.3.0 seems to violate this seemingly basic heads up but i didnt review this issue close enough to know that my assertion here is correct. if it is i dont know what semver is for anymore.

apologies if this seems strident i dont mean it to be. i had some intensive rollbacks i had to do where i was used to trusting these upgrades without doubt.

anyway, its mostly a pycharm issue i agree but a breaking change is a breaking change. my limited perspective says that the built in warning systems did not work.
On Dec 1, 2020, 2:14 AM -0800, Bernát Gábor [e-mail], wrote:

Should this critical breaking change be coordinated between the biggest IDE's developers and the developers of the most important executable in the python ecosystem? Probably.
All major IDE developers have access to the pip changelog. The pip developer team has no way to know what deprecated interfaces these IDEs use (against emitting warnings for 6+ months), especially sometimes when those sources are behind closed doors. Yes, you should be opening a PyCharm issue. The issue lies with them at the end of the day. Should they perhaps have a test suite against the master branch of pip? Probably.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[gaborbernat]

That said maybe this should have been a semver major release change on the pip side?

If you take a look at pip version number (2020.4) you'll notice that's a fairly big number. It's because pip does not use semantic versioning but instead https://calver.org/overview.html. As such there's no concept of major release you might be familiar with semver.

it should be noted that a user has to adopt this pip change to break things and it can be rolled back in the same way.

The warning message printed on every invocation for the last few months (at least) should suffice as a notification of you're doing something wrong. Please familiarize yourself with it https://calver.org/overview.html#when-to-use-calver 👍

[pradyunsg]

That said maybe this should have been a semver major release change on the pip side?

If we followed Semver -- sure, it would've been -- but we don't, for a lot of reasons that I won't go into. I'll flag that pip has been one of the most prominent projects using CalVar for over 2 years now.

We also have a documented deprecation policy about making breaking changes: https://pip.pypa.io/en/stable/development/release-process/#deprecation-policy

my limited perspective says that the built in warning systems did not work.

This is definitely true. I do want us to identify why this happened, so that we can avoid this class of issues in the future.

---

I say that we let this thread be "at rest" now, until tomorrow or until one of the PyCharm developers responds here. :)

[east825]

Hi everyone! I'm from a PyCharm team. We didn't notice the deprecation because this option was used only in an internal script that we normally don't launch manually. We can definitely improve in this regard by reporting any deprecation messages from the tool to stderr, at least in the internal/EAP mode as an early diagnostic, so that it doesn't happen again.

As for why we needed --build-dir in the first place, it seems that now it's more of a historical artifact back from the days when pip didn't build binary packages in TMP by default, so we had to emulate this behavior (creating a temporary directory, building in it and then removing) manually. Unfortunately, the history is somewhat scarce, and the option was added in 2013. Can someone shed the light on what was the former behavior, so that we were entirely sure that by removing the option we don't break some corner cases? In the distutils-sig thread that I started Tzu-ping Chung has already mentioned that this behavior changed around pip 6.x. Is it accurate? Would you mind pointing me to the corresponding changelog items?

Making --build-dir a no-op for a little while, until we prepare updates to already released versions of PyCharm, especially if we really don't need it any longer, sounds just perfect.

[mgedmin]

A modest proposal: if you add a time.sleep() after printing the deprecation warning, people will start investigating why pip is suddenly slow and will notice the warning.

For best results increase the delay with every subsequent release.

[gaborbernat]

For best results increase the delay with every subsequent release.

This can be interpreted as one of those passive-aggressive interactions 🤔 What about people who can't upgrade and pin, should they be penalized?

[brainwane]

@mgedmin I presume that your phrase "modest proposal" was meant to mark your comment as satirical, and not to be taken seriously. Am I correct?

[Also, @east825, thank you & thanks to PyCharm for [sponsoring the Python Software Foundation](https://pyfound.blogspot.com/2020/12/help-psf-raise-60000-usd-by-december.html) and helping financially contribute to the infrastructure of Python!]

[east825]

@brainwane I've passed unexpected, but heart-warming praises to the team :)

[east825]

@pradyunsg Would it be possible to keep the dummy no-op option for another 6 months (maybe even hiding it in the --help output)? We'd like to ensure that most of the users of 2020.x releases managed to upgrade to the patched versions of the IDEs.
I realize that it's not at all "a little while", but it would help us a lot with a smooth transition.

[sbidoul]

@pradyunsg @east825 I'll do the PR within the next hours.

[pradyunsg]

Would it be possible to keep the dummy no-op option for another 6 months (maybe even hiding it in the --help output)? We'd like to ensure that most of the users of 2020.x releases managed to upgrade to the patched versions of the IDEs.

Yes, I'm 100% on board for doing this.

I'm pretty sure that no one in @pypa/pip-committers would be opposed to this, but, if they are, I'd imagine we'll hear from them here. ;)

@sbidoul wheee! Thanks! :)

[sbidoul]

There you go: #9198.

[pradyunsg]

FYI folks - I plan to keep this issue open, until the re-removal of the option in 2021. That way we can cross check before 21.1 that PyCharm folks have been able to successfully transition most of their users.

[SergeiBond]

Can this error break venv setup in any way?

[east825]

@SergeiBond It can interfere, if you're automatically installing a package in it, e.g. Django or Flask, through a new project wizard. An empty virtual environment is still created in this case, though.

[sfelixkim]

It looks like the new release of PyCharm (2020.2.5) got it fixed? PyCharm people, please confirm :)

[east825]

@sfelixkim Correct! 2020.2.5 was released specifically because of this fix. It's also included in 2020.1.5 and the upcoming 2020.3.

[pradyunsg]

I'll cut a release in my lunch break today. :)

[east825]

First of all, huge thanks to everyone involved in addressing this problem so promptly!

I don't want to be a pain in the neck, but, nonetheless, can someone please still elaborate a bit on how the behavior of install with and without --build-dir has changed over the course of the years, so that the option has become effectively redundant and it was decided to drop it altogether?

[pradyunsg]

I'm not a 100% sure on the history, but based on what you described earlier, yes, it's redundant for PyCharm's use case now. :)

I've listed what seem to be the changelog entries relevant to this option (from https://pip.pypa.io/en/latest/news/). Folks can Ctrl+F to figure out the timeline and all that. :)

• Deprecate -b/--build/--build-dir/--build-directory. Its current behaviour is confusing and breaks in case different versions of the same distribution need to be built during the resolution process. Using the TMPDIR/TEMP/TMP environment variable, possibly combined with --no-clean covers known use cases. (#8372)
• Use a randomized and secure default build directory when possible. (#1964, #1935, #676, #2122, CVE-2014-8991)
• DEPRECATION pip install --build and pip install --no-clean are now NOT deprecated. This reverses the deprecation that occurred in v1.5.3. (#906)
• DEPRECATION pip install --build and pip install --no-clean are now deprecated. (#906)
• Added documentation on usage of --build command line option (#4262)

[vurbl]

To imply that pip's devs didn't do enough feels exceedingly dismissive of our efforts, and is the kind of thing that results in folks walking away from projects.

Agreed.

Can I also note that the comments I saw from the PyCharm developers were polite, reasonable, and not at all confrontational, so I'd like to publicly thank them for taking a completely reasonable approach here.

Drive-by criticism of either project over this from 3rd-party bystanders is neither helpful nor welcome, in my view.

Apologies, thanks for the vigorous response. Truly appreciated.

[uranusjr]

I took the liberty to create the 21.1 milestone add this to it.

[pradyunsg]

Thanks @uranusjr! FWIW, 20.3.1 is out with the requested fix.

[SergeiBond]

@SergeiBond It can interfere, if you're automatically installing a package in it, e.g. Django or Flask, through a new project wizard. An empty virtual environment is still created in this case, though.

Hmm, weird: I got this error when I was actually starting a new Flask project with venv. However, everything seems to be working fine despite the error. It seems that Flask package got installed just fine.

[east825]

@pradyunsg Thanks a lot for the excerpt from the changelog. Apparently, #2122 is just what I was looking for. Shame on me, somehow I couldn't find it myself right away. All in all, the option indeed seems long overdue for removal from our scripts.