Conda-forge.github.io: pypy next steps

Created on 18 Sep 2019  Â·  68Comments  Â·  Source: conda-forge/conda-forge.github.io

TODO:

Then, start building packages

picture ericdill

Most helpful comment

I've added PRs to all of the TODOs in the first comment. They need to be merged in order.
Locally, I can now switch the python interpreter to pypy and noarch: python recipes work.

picture isuruf on 11 Feb 2020
🎉3 👍2

All 68 comments

Hi. I believe this is a follow up to conda-forge/conda-forge-pinning-feedstock#225 which proposed (as far as I understand it, I may be wrong):

  • modify recipe/conda_build_config.yaml to add pypy tags
  • fix any breakage for parsing the longer tag
  • add required pypy versions somewhere
  • rebuild all packages with this tag.
mattip picture mattip on 18 Oct 2019

I think there are some things on the migration/bot side that need to be done (not that this is blocking the other steps except for the rebuild).

  • A way to have the bot update the conda_forge_config.yaml
  • A way to scope migrations (in case we don't want all the packages to be rebuilt)

Aside from these the migration will look like the arch rebuild.

CJ-Wright picture CJ-Wright on 18 Oct 2019

Once the Python 3.8 rollout activity dies down, I would love to move this issue forward.

mattip picture mattip on 2 Nov 2019
👍1

@CJ-Wright @ocefpaf ping

mattip picture mattip on 18 Nov 2019

@mariusvniekerk do you think it better to enhance the current cfep9 migration spec in the bot or make a bespoke migrator class?

CJ-Wright picture CJ-Wright on 18 Nov 2019

@mattip do you have a list of packages that you want to build this out for? You don't need to list everything, we can get the needed deps for the packages you want to rebuild.

CJ-Wright picture CJ-Wright on 18 Nov 2019

numpy, scipy, pandas, matplotlib would be a great start

mattip picture mattip on 18 Nov 2019

Say yes / no to the spec, point out potential pitfalls

Anyone got a link to the mentioned "spec"?

mbargull picture mbargull on 18 Nov 2019

Perhaps the reference is to this meeting's minutes which reference this plan

mattip picture mattip on 18 Nov 2019

Also: dask, streamz, and xonsh. Are there issues with the library search order and noarch support?

CJ-Wright picture CJ-Wright on 19 Nov 2019

I put together this slightly more fleshed-out plan: https://github.com/msarahan/pypy-python-structure

Compared to the earlier plan, it is much less disruptive in terms of hotfixes. However, renaming the python package is rather dramatic (though completely reasonable in my opinion) and this probably needs a lot of community review.

I hope it's enough to help you push forward.

msarahan picture msarahan on 19 Nov 2019
👍1

I commented there. How does this affect user's experience? Would a user ever get into a situation where they have two interpreters in the same conda environment?

mattip picture mattip on 20 Nov 2019

A couple of questions.

  1. Can we get rid of _python_impl (which enforces that two interpreters are not in the same environment) and use run_exports to do the same. This might reduce confusion as it results in only 2 packages instead of 3.
  2. Do we want to hotfix existing packages to enforce cpython or allow existing packages work with cpyext?
isuruf picture isuruf on 20 Nov 2019

allow existing packages work with cpyext

I am not sure what this means. Pure python packages will work as-is. C-extension packages will need to be rebuilt with the relevant PyPy interpreter each time PyPy releases a new major.minor version, since the c-extension modules built with CPython cannot be loaded into PyPy and visa-versa. In fact. each time PyPy changes its major.minor version, the modules will need rebuilding. Those changes should happen less frequently now that cpyext is beginning to stabilize. Note that PyPy versions have two components: the python side, which is compatible with CPython and reflected in sys.version_info, and the cpyext ABI side, which is reflected in sys.pypy_version_info. PyPy's last release was pypy_verision_info == 7.2.0 and version_info 2.7.13 and 3.6.9 (matching CPython2.7.13 and CPython3.6.9). The next, which should happen by the end of the year, will increment the pypy_version_info to 7.3.0.

mattip picture mattip on 20 Nov 2019

Ah, I must have read about cpyext wrong. Thanks for the info.

isuruf picture isuruf on 20 Nov 2019

@mattip, thanks for the link to the Google Doc.
@msarahan, thanks for the refined version. That one is much more in line with what I had in mind (cpython and pypy).

I commented there.

xref: https://github.com/msarahan/pypy-python-structure/commit/7b71aed2b6aa752f08654e9fd86cdeebe78a8f4c#r36041947 (just so that those comments don't get lost)

When I have further questions, I'll post them at https://github.com/msarahan/pypy-python-structure/issues

mbargull picture mbargull on 20 Nov 2019

Can we get rid of _python_impl (which enforces that two interpreters are not in the same environment) and use run_exports to do the same. This might reduce confusion as it results in only 2 packages instead of 3.

I don't think so, unfortunately. There must be a metapackage above to provide the users the choice of implementations and to stand in for the existing references to "python". That can't serve as the mutex, because then you'd have a cyclical dependency. So, you also need a layer below that also converges on one name (that's the mutex part). If you see a better way, I'm all for it, but I think 3 packages is the minimum.

msarahan picture msarahan on 20 Nov 2019

Compared to the earlier plan, it is much less disruptive in terms of hotfixes

If I understand @msarahan's correctly, all the existing python packages would need to be hotfixed to add a dependency on a cpython mutex metapackage. I believe when this is deployed all existing conda environment containing python would suddenly become inconsistent and the solver would be free to install just about anything. I believe that adding the the cpython metapackage as a constraints would accomplish much of the same behavior without creating inconsistent environment.

Similarly, adding the cpython metapackage to constraints to packages with compiled extensions would prevent them from co-existing with pypy assuming it required a conflicting metapackage.

Along these lines I think it is possible to retain the name of the python package while allowing a lower priority python metapackage which requires pypy. The only hotfixing would be adding constrains entries to. I'll put together an proposal.

jjhelmus picture jjhelmus on 20 Nov 2019

Noticed that @mbargull made the same observation about inconsistent environment in msarahan/pypy-python-structure#3.

jjhelmus picture jjhelmus on 21 Nov 2019

Yes, adding hard dependencies via hotfixing is a too disrupting thing, esp. when we are talking about one of the most common packages, python.
Generally, I would prefer a solution that wouldn't require changing the metadata of nearly every single python-dependent package. But if we want to keep the current dependency scheme, I agree that adding constrains on the Python implementation to the python-dependent packages seems like the best working compromise.
Depending on the naming/topology of the Python implementation packages, that constains would either need to be preclusive, e.g., "constrains": [ "pypy <0a0" ], or make things more explicit, e.g., "constrains": [ "python_impl=*=*_cython" ].

mbargull picture mbargull on 22 Nov 2019

Here's a less intrusive plan,

  1. Add "constrains": [ "pypy <0a0" ] to all existing python packages and python dependent packages.
  2. Change the build strings of new python packages with _cpython at the end. (Keep the old python packages as is).
  3. Add a run_exports of python * *_cpython to the python packages.
  4. Add a python 2.7 *_7_1_pypy package which has run_exports of python * *_7_1_pypy and it would depend on the pypy package.

With this, no existing recipes need to change.
noarch: python packages will also be dependent on cpython packages. This is needed until conda-build adds support for pypy as well.
When conda-build adds support, then we can rebuild the noarch: python packages with build: ignore_run_exports: - python.

2, 3 - https://github.com/conda-forge/python-feedstock/pull/309
1 - https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/27

Above two PRs can be merged as is, if you agree.

isuruf picture isuruf on 2 Feb 2020

Also, defaults need to do 1 as well, but they don't need to do 2, 3, 4 for conda-forge to go ahead.

isuruf picture isuruf on 2 Feb 2020

Building for pypy would look like https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/225/files

isuruf picture isuruf on 3 Feb 2020

@mbargull, can you have a look at https://github.com/conda-forge/conda-forge.github.io/issues/867#issuecomment-581083002 ?
I basically took yours and @msarahan's proposal and merged python_impl and python into one package as there isn't any reason to have two.

isuruf picture isuruf on 3 Feb 2020

Depending on _cpython in the build strings would create a split in packages. Those build after the new package would not work with the older python packages because of the build string requirement.

Rather than using build string for the mutex why not add a constrains on the python implementation type via a selector package?

Specifically:

  1. Hotfix the existing python packages and all Python using packages using with a "constrains": [ "python_impl * cpython" ]
  2. Hotfix pypy to add a python_impl *_7_1_pypy to depends.
  3. Add a run_exports of python_impl * *_cpython to the python packages.
  4. Add a python 2.7 package which has a run_exports of python_impl * *_7_1_pypy and it would depend on the pypy package.

This allows hotfixing in the future and the python_impl package can be used to select which Python implementation to use.

jjhelmus picture jjhelmus on 3 Feb 2020

I suppose there is also:

  1. Create python_impl * cython and python_impl * *_7_1_pypy metapackages packages.
jjhelmus picture jjhelmus on 3 Feb 2020

Make sense. I'd still like to add _cpython to build string so that we can make sure they are used when building new packages.

What about versioning of python_impl? major.minor or major.minor.patch?

isuruf picture isuruf on 3 Feb 2020

We can have python_impl=7.1.0=pypy and python_impl=3.7.6=cpython.

isuruf picture isuruf on 4 Feb 2020

We can have python_impl=7.1.0=pypy and python_impl=3.7.6=cpython.

Sorry for the long reply, but I am not familiar with the conda package versioning and want to make sure we are talking about the same things.

7.1.0 relates to the abi tag. The python tag for pypy is pp27 for python 2.7 and pp36 for python 3.6. Wheels are marked with a python tag, an abi tag, and a platform tag.
For cpython this looks like numpy-1.19.0-cp36-cp36m-linux_x86_64.whl where

  • cp36 is the python tag,
  • cp36m the abi tag.
    For PyPy, this looks like numpy-1.19.0-pp36-pypy36_pp73-linux_x86_64.whl (using latest versions of pip>20 and wheel>0.34.2), where
  • pp36 is the python tag,
  • pypy36_pp73 is the abi tag.
    Both use linux_86_64 for the platform tag.

Which is a long way of saying python_impl=7.1.0=pypy is both over precise (no need to include the patch number) and somewhat confusing (conflates the abi tag with the python tag, so pp27 and pp36 would both have the same build string). What about using the ABI tag for the python_impl?

I think there is no need for the patch number in any of the build strings: if the ABI changes it should change the minor number, and the same with the python tag.

mattip picture mattip on 4 Feb 2020

Make sense. I'd still like to add _cpython to build string so that we can make sure they are used when building new packages.

Agreed, having the build string and a python_impl package makes it clear and allows two methods for selecting the implementation:
conda create -n test python=3.8*=*cpython*
conda create -n test python=3.8 python_impl=*=cpython

What about versioning of python_impl?

That makes sense, although the solver does consider the version ordering when optimizing the solution so we need to keep that in mind. Putting the versioning in the build string is less effect on conda. I like @mattip suggestion of using the ABI tag but that will likely need to go into the build string rather than the version.

jjhelmus picture jjhelmus on 4 Feb 2020

See https://github.com/conda-forge/staged-recipes/pull/10760
I'm creating packages like,

  • python_abi=3.6=cp36m
  • python_abi=3.6=pypy36_pp73
isuruf picture isuruf on 4 Feb 2020

@mattip, what versions of pypy do you want to support?
Currently we have

  • pypy2.7 - 7.1.0 on linux-64 and osx-64 (no 7.3.0 yet and no linux-aarch64, linux-ppc64le, win-64)
  • pypy3.6 - 7.3.0 on linux-64 and osx-64 (no linux-aarch64, linux-ppc64le, win-64)

Can you send PRs to update the above recipes?

isuruf picture isuruf on 4 Feb 2020

@mbargull, can you have a look at #867 (comment) ?

I haven't yet/can't look into it right now or tomorrow, sorry. Ping me again if I haven't answered by Friday.
Quick question: Which packages will have which content? Meaning, which package will hold the actual implementations? Still pypyX.Y and python * *_cpython or the _impl packages? That confused me a bit (but I'm also not perceptive right now..)

mbargull picture mbargull on 4 Feb 2020

Still pypyX.Y and python * *_cpython packages. Others will be meta-packages and have no content.

isuruf picture isuruf on 4 Feb 2020

Can you send PRs to update the above recipes?

The latest would be pp27_pypy73 and pp36_pypy73 on linux32, linux64, aarch64, ppc64le, s390x, and windows32bit. I realize the latter is not supported by CPython, but PyPy does not support 64-bit windows so I guess that means no windows at all. Which repo/pr needs my input? conda-forge/conda-forge-pinning-feedstock#225 ?

mattip picture mattip on 4 Feb 2020

https://github.com/conda-forge/pypy3.6-feedstock/
https://github.com/conda-forge/pypy2.7-feedstock/

win32 requires building the dependencies with win32 and I don't think we will go back to doing that.

isuruf picture isuruf on 4 Feb 2020

Given that we are in 2020 and it would take time to port an older version of numpy to pypy 7.3 (and other packages), I suggest dropping python 2.7 for pypy.

isuruf picture isuruf on 4 Feb 2020

@msarahan, @jjhelmus, @mbargull, how hard would it be to fix conda-build to automatically add ignore_run_exports: - python to noarch: python recipes? This is to avoid the cpython or pypy specific run_exports added.

isuruf picture isuruf on 5 Feb 2020

Probably not too hard. You could do it at the tail end of rendering, when
the metadata objects for each output get finalized.

On Tue, Feb 4, 2020, 18:02 Isuru Fernando notifications@github.com wrote:

@msarahan https://github.com/msarahan, @jjhelmus
https://github.com/jjhelmus, @mbargull https://github.com/mbargull,
how hard would it be to fix conda-build to automatically add ignore_run_exports:

  • python to noarch: python recipes? This is to avoid the cpython or pypy
    specific run_exports added.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/conda-forge/conda-forge.github.io/issues/867?email_source=notifications&email_token=AAAJL6LTBH735URGY4H74NLRBH6Y3A5CNFSM4IYBZS62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKZUZZQ#issuecomment-582175974,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAJL6LENWFGJAMBRPU6GQTRBH6Y3ANCNFSM4IYBZS6Q
.

msarahan picture msarahan on 5 Feb 2020

I've sent conda/conda-build#3868
I'm waiting on a decision on https://github.com/conda-forge/staged-recipes/pull/10760 to go forward.

isuruf picture isuruf on 5 Feb 2020

Next blocker https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/27

isuruf picture isuruf on 5 Feb 2020

Next blocker is https://github.com/conda/conda-build/pull/3868

isuruf picture isuruf on 8 Feb 2020

Also needs a decision on conda-forge/python_abi-feedstock#2

isuruf picture isuruf on 8 Feb 2020

I've added PRs to all of the TODOs in the first comment. They need to be merged in order.
Locally, I can now switch the python interpreter to pypy and noarch: python recipes work.

isuruf picture isuruf on 11 Feb 2020
🎉3 👍2

@jjhelmus, @mingwandroid, looks like hashing ignores the python pinning and explicitly uses major minor pinning at https://github.com/conda/conda-build/blob/391d5326aaffbf5d30f4baf908d58c77c58ee878/conda_build/metadata.py#L1260-L1264
How about using the contents of keys like python if they have a space in them? That should be backwards compatible right?

isuruf picture isuruf on 4 Mar 2020

Sounds fine to me yes.

mingwandroid picture mingwandroid on 4 Mar 2020

If anybody is interested, you can try out by doing,

conda config --set add_pip_as_python_dependency False
conda config --set channel_priority strict
conda create -n pypy pypy

You can't do conda install numpy in this env because numpy is not built.
You can however do conda install mpmath because mpmath is a noarch package.

isuruf picture isuruf on 4 Mar 2020

Hey all, we are seeing some issues where PyPy is being pulled in for Python 3.6 where CPython was pulled in before. Am a little unclear if this is expected or if we should be doing something differently. Thoughts? 🙂

Edit: Here's a no-op PR ( https://github.com/conda-forge/distributed-feedstock/pull/118 ) with a build with demonstrating this issue.

jakirkham picture jakirkham on 5 Mar 2020
👀2

Proposing pulling the new Python 3.6 packages until someone has time to debug this further. ( https://github.com/conda-forge/cf-mark-broken/pull/15 )

jakirkham picture jakirkham on 5 Mar 2020
👍1

FWIW at a package level, this seems to be a viable workaround. Not the prettiest thing though

# filename: recipe/meta.yaml

package:
  name: blah
  version: 1

requirements:
  host:
    - python
    - python_abi * *_cp36m  # [py36]
  run:
    - python
    - python_abi * *_cp36m  # [py36]
jakirkham picture jakirkham on 5 Mar 2020

Maybe this is the missing piece ( https://github.com/conda-forge/python-feedstock/pull/321 )?

Edit: Looks like Python 2.7 may need this too ( https://github.com/conda-forge/python-feedstock/pull/322 ).

jakirkham picture jakirkham on 5 Mar 2020

Also it seems like these changes were not applied to Python 3.7. Is that expected?

jakirkham picture jakirkham on 5 Mar 2020

First PR to add a pypy build https://github.com/conda-forge/certifi-feedstock/pull/53

isuruf picture isuruf on 9 Mar 2020

Thanks for all the tireless work on this, @isuruf!

h-vetinari picture h-vetinari on 9 Mar 2020
❤1

Last 2 PRs. conda-forge/setuptools-feedstock#134 and https://github.com/conda-forge/conda-forge-ci-setup-feedstock/pull/70

isuruf picture isuruf on 10 Mar 2020

Closing now. See https://conda-forge.org/docs/maintainer/knowledge_base.html#pypy-builds for docs

Only remaining issue is patching defaults. To work around that use strict channel priority as indicated in the docs.

isuruf picture isuruf on 10 Mar 2020
🎉2

Thanks @isuruf! 😄

jakirkham picture jakirkham on 10 Mar 2020

Thanks again to the conda forge team for the buildout of PyPy packages.

PyPy has released 7.3.2 which is binary-compatible with 7.3.1.
Can/should we move forward with

  • updating the base package with 7.3.2
  • windows 32 bit PyPy builds. PyPy does not yet offer 64-bit windows. Should this wait for 64 bit windows?
  • PyPy now offers python 3.7: could we roll out a 3.7 offering in parallel to the 3.6?

Is there a process flow for doing any of the above?

mattip picture mattip on 26 Sep 2020

That makes sense. Yeah we dropped Windows 32-bit support a long time ago. So I think we would need to wait for 64-bit support for Windows. Adding Python 3.7 PyPy would be great!

Updating the existing Python 3.6 PyPy, should just be a matter of adding a PR to this feedstock. Adding Python 3.7 would be done through a new recipe to staged-recipes, which could be based of the pypy3.6 recipe. We would then want to either update or add a new migration based off of the existing one for pypy.

If anyone else sees something we would need to do here, please hop in :)

jakirkham picture jakirkham on 26 Sep 2020

What is the support plan for pypy3.6? Should we build it with pypy3.7?

CJ-Wright picture CJ-Wright on 26 Sep 2020

For the time being that sounds like a good idea. It's the only PyPy version that we have a decent number of packages built for.

jakirkham picture jakirkham on 27 Sep 2020

This is the ultimate proof of readiness for PyPy and it's time that we could simply use it they have put so much work into it over the last 15 years can you guys automate kicking off builds of everything for PyPy?

brianmingus2 picture brianmingus2 on 9 Jan 2021

@brianmingus2 My reply may be inappropriate, please clarify your intention if I missed.

This issue describes the extensive work conda forge completed to make PyPy3.6 available. On Sept 26 I used the closed issue to bring up the new release of PyPy3.7, I probably should have opened a new issue instead. In the mean time, the migration for PyPy3.7 has begun. Some 3.7 packages are already available, more are coming.

mattip picture mattip on 9 Jan 2021

Thank you for the reply, what is the hangup, lack of integration tests requiring manual review?

brianmingus2 picture brianmingus2 on 9 Jan 2021

See the conda forge status page:

conda-forge.org/status

It lists the packages that are done, in pr etc. help fixing failed builds is appreciated!

beckermr picture beckermr on 9 Jan 2021

The PyPy3.7 migration only began 4 days ago. It takes a while for the bot to churn through all the packages.

mattip picture mattip on 9 Jan 2021
👍1

The pypy guys just released an updated 3.7 version that now includes Windows 64-bit support. Is there something that needs help to get this integrated?

asteppke picture asteppke on 16 Apr 2021

@asteppke see conda-forge/pypy3.6-feedstock#39. We need to expand the recipe to download the windows64 pypy2.7 in order to build pypy3.7

mattip picture mattip on 16 Apr 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

scopatz picture scopatz  Â·  4Comments
artPlusPlus picture artPlusPlus  Â·  5Comments
bulli92 picture bulli92  Â·  4Comments
Cadair picture Cadair  Â·  5Comments
jakirkham picture jakirkham  Â·  5Comments