Conda-forge.github.io: Supporting non-standard CPython VC builds on Windows

[jakirkham]

The current system of having the version Windows' MSVC tied to the CPython is troublesome for people in some cases. In particular, with respect to cases where people needed the latest compiler features (e.g. C++11 features). There may be other relevant cases, as well. The main purpose of this issue is to suggest that we include some (maybe even just one) compilation of a non-standard CPython VC build on Windows to simplify things for the conda community. Of course, making this change could disrupt the current feature landscape of Windows CPython so needs some thought on that point. This issue is also opened to get feedback from the conda community to determine which non-standard CPython VC builds would be valuable and thus be worthwhile to support. At present, I am thinking a vc14 (i.e. Visual Studio 2015) variant of CPython 2.7.x (where x will always be latest) is almost certainly worthwhile just to get C++11 support. Though am not sure if there are other worthwhile non-standard variants to consider.

cc @ukoethe @jasongrout @SylvainCorlay @msarahan @mcg1969

[msarahan]

This obviously comes with a huge YMMV, but would be helpful to a lot of people. You could have installers like what @jjhelmus did with ACPD: https://github.com/acpd/acpd

A lot of people will probably get very confused by incompatible software, but we just need to document what this is and is not very, very clearly.

[jakirkham]

Absolutely. This would also be a very hands on endeavor that would really need community involvement to be successful.

[jakirkham]

Once we nail down which non-standard variant could be reasonably supported here and what accommodations may be needed in conda/conda-build. We may need to revisit the AppVeyor build matrix to include this logic there. That way, we can do a better job at providing things that will play nice together.

[ChrisBarker-NOAA]

This is a very big deal, actually. Not to be taken lightly at all!

key is that it would essentially be a different platform. You'd have Windows 32 and 64 bit (confusing enough for some folks), and now multiple versions of those. Though I suppose we'd really only need to support Py27 on Windows64. Py3.5 is already using MSVS2015.

But one of the nice things about conda (and Anaconda) is that the binaries are compatible with the python.org ones, so that pip install also works. We'd have to make sure that someone running the MSVS2015 build couldn't accidentally get an incompatible binary from PyPI, (Or anywhere else).

Also -- we'd then need to build everything in the default channel, too.....

-CHB

[jakirkham]

This is a very big deal, actually. Not to be taken lightly at all!

Agreed. That being said there are many people that are frustrated by not having this. So, I would really like us to think about solving it somehow. Whether it be the way suggested above or adding some other sort of compilation or something else entirely.

key is that it would essentially be a different platform....

Honestly, I am a little fuzzy on how this would work, in particular, when looking at the feature space. Right now we have a monotonic function of Python to VC. This change would break that. Not only would it not be monotonic, but it would not be a function. So, really worth some thought.

Though I suppose we'd really only need to support Py27 on Windows64.

That's another trick that we have to adjust for.

We'd have to make sure that someone running the MSVS2015 build couldn't accidentally get an incompatible binary from PyPI, (Or anywhere else).

Hmm...that is a good point. Thinking about wheels here? Maybe we can disallow them entirely for a non-standard Python. It might get some grumbles, but then again just add a recipe.

Also -- we'd then need to build everything in the default channel, too.....

It was sort of my impression that we are doing this anyways. So, this might be the one part that is not a problem. The only part to work on here would be to add this to our Windows build matrix. Though this is many steps later.

[ChrisBarker-NOAA]

key is that it would essentially be a different platform....
Honestly, I am a little fuzzy on how this would work, in particular, when looking at the feature space. Right now we have a monotonic function of Python to VC. This change would break that.

well, there are a number of platforms -- one of them is win-64 -- due to its Python roots, I assume that conda assumes that that means the VS2008 run-time C lib as well -- obviously for Python, but also for any shared libraries. So I think the way to do this would be to create a new platfrom, not another python variant.

win-64-15 ????

I have no idea how hard-coded the various platforms are in the whole conda / anaconda ecosystem -- is it hard to add a new one?

(also, it would be a bit unfortunate, as pure python libs, etc would all need separate builds...)

But I just had a realization -- py2.7 and py3.5 currently use different run times, so shared libs that re going to be used with python should be built differently -- how in the world is that dealt with now???

-CHB

[ChrisBarker-NOAA]

We'd have to make sure that someone running the MSVS2015 build couldn't accidentally get an incompatible binary from PyPI, (Or anywhere else).
Hmm...that is a good point. Thinking about wheels here? Maybe we can disallow them entirely for a non-standard Python. It might get some grumbles, but then again just add a recipe.

I don't know how hard / possible it is to disallow binary wheels with a given pip install (and someone could install pip themselves from source, etc....)

probably what you'd want to do is make sure the platform tag used by python (and therefor found by pip) was unique.

But The core python devs are not going to build a Python with a new MS compiler because of all this pain, and because py2.7 is a dead end -- it has always worked better to simiply use a newer compiler when you have a newer version --a nd there is not going to be a newer version of 2.7.

IN fact, the stackless folks were pushing for a 2.8 that was the same except for the MS compiler used to build it -- that was not accepted as a reasonable option.

So maybe it's time to just say -- if you want modern C++, then you should use a modern Python, too.

I know, tough love!

[ukoethe]

@ChrisBarker-NOAA

So maybe it's time to just say -- if you want modern C++, then you should use a modern Python, too.

Life is not that easy. Consider, for example, our ilastik project: it requires over 50 dependencies, including ~40 compiled extension modules. Some of them require modern C++, others lack support for Python 3 (or just Python 3.5). Most of these packages supply scientific algorithms that are not at all easy to reimplement. How would you solve the dilemma?

I ended up compiling Python 2.7 and everything else using Visual Studio 2012. Adapting Python to the new compiler was actually very easy: I just had to inform distutils about this compiler version (in only one place), remove/update three outdated compiler options, upgrade the sqlite version, and lift the build to 64-bit. See https://github.com/ilastik/ilastik-build-conda/tree/toolset-config/python for details.

key is that it would essentially be a different platform

No, this problem can be solved with features (subject to a fix in the version resolution algorithm that went into conda 4.0). Suppose there is a package vc11-runtime that contains a declaration track_features: - vc11. Then, users can tell conda to use Visual Studio 2012 by simply installing that package first:

conda create  -n python-vc11  vc11-runtime  python=2.7 ...

conda will now prefer package variants that contain a features: - vc11 declaration over those that don't. So, simply put that declaration into packages compiled with vc11, and let conda figure out the rest.

[jakirkham]

If we use an even more recent compiler for this sort of endeavor, would that still work for you, @ukoethe?

[ukoethe]

If we use an even more recent compiler for this sort of endeavor, would that still work for you?

Yes, but I haven't checked if there are any hidden difficulties along the way.
(For example: I use devenv PCbuild.sln /upgrade to update Python's official project files to the new compiler version. Will this command still work as desired for VS 2015?)

[ukoethe]

That the binaries are compatible with the python.org ones, so that pip install also works.

This is indeed a problem when using a non-standard compiler. However, one could easily add a warning to conda's pip module saying that pip downloads may be incompatible with the current environment when a non-standard compiler was used.

On the other hand, conda _exists_ because pip's support for compiled modules with complicated C/C++ dependencies was insufficient. People familiar with conda will probably use pip only for simple things like pure Python modules (where the compiler doesn't matter), and wouldn't be surprised if pip failed for more complicated stuff.

[ChrisBarker-NOAA]

Some of them require modern C++, others lack support for Python 3 (or just Python 3.5).

yeah, I guess that's the trick -- I would be more or less happy to tell anyone writing a package that needs C++11 to use py3, but then you've got other stuff you want to use that isn't py3 ready ( :-( ). I sure wish everyone used Cython -- then the py3 port would be pretty easy.

Honestly, though I still think we'l all need to go to py3 some day -- the more reasons to support that the better.

[ChrisBarker-NOAA]

key is that it would essentially be a different platform

No, this problem can be solved with features (subject to a fix in the version resolution algorithm that went into conda 4.0). Suppose there is a package vc11-runtime that contains a declaration track_features: - vc11. Then, users can tell conda to use Visual Studio 2012 by simply installing that package first:

conda create -n python-vc11 vc11-runtime python=2.7 ...

conda will now prefer package variants that contain a features: - vc11 declaration over those that don't. So, simply put that declaration into packages compiled with vc11, and let conda figure out the rest.

I still don't "get" features, but if so -- great.

but it sounds risky -- conda will prefer packages with the feature, but I take it it will allow packages that don't have it -- which is the vast majority of compiled packages -- so it would be very easy for a user to get an incompatible package -even without pip.

It still seems to me that you'd want this to be a different platform -- or maybe a different python version.

call it py28 -- against the wishes of the core devs :-)

But I'll leave that to the conda experts -- you may be right.

[jakirkham]

@patricksnape, I forgot to cc you on this. Sorry about that. Maybe of some interest to you.

[patricksnape]

Although my gut is that I'm against this :stuck_out_tongue_closed_eyes: I understand @ukoethe's pain about scientific packaging. Ideally the response would be to upgrade those projects so that they work on Python 3.5 and then use Python 3.x so you get a compliant compiler. However, as I said, I totally understand that we are all busy and that is often impractical. Though someone will have to do it before 2019!

I would throw my weight behind the features option - since any code without compiled extensions doesn't actually care about the Visual Studio version that was used to build it. Any conda-forge code should have the correct feature set anyway so that won't be a problem. I strongly urge against Python 2.8.

In terms of pip, as @jakirkham mentioned, the whole point of conda was to try and standardise a lot of these issues. Given the lack of community around providing wheels, I doubt that a lot of wheels being provided by anyone other than Christoph Gohlke are actually being careful about which compiler is used. Therefore, I would say that we document against using wheels and always prefer the pip install without wheels option when using conda on Windows.

[ukoethe]

I would throw my weight behind the features option

Another important point in support of this option is that features (or some yet-to-be-defined alternative) are needed for a number of other legitimate configurations anyway (Python version, BLAS variant, AVX acceleration, ...). Then, adding a compiler feature is not such a big deal -- it only makes the build matrix a bit bigger and requires build scripts to be a bit more portable. I don't expect this to be very complicated once the first compiler on a given platform works.

Of course, I agree that in an ideal world, all packages should support the newest Python version.

[jakirkham]

Another important point in support of this option is that features (or some yet-to-be-defined alternative) are needed for a number of other legitimate configurations anyway (Python version, BLAS variant, AVX acceleration, ...).

That's a whole other issue. To be precise this issue ( https://github.com/conda-forge/conda-forge.github.io/issues/49 ). If we can avoid commingling them, that would probably be best for everyone's sanity. Though it is true that lessons from one can likely be applied to the other.

I don't expect this to be very complicated once the first compiler on a given platform works.

We already use different compilers with Windows. My concern from earlier involves changes to the meta.yaml, which I know you have already spent some time thinking about. Maybe worth sharing. :wink:

[ChrisBarker-NOAA]

Another important point in support of this option is that features (or some
yet-to-be-defined alternative) are needed for a number of other legitimate
configurations anyway (Python version, BLAS variant, AVX acceleration, ...).

Things like BLAS are only required by, well, things that require BLAS.

The compiler used (or, more to the point, the runtime used) effects every
compiled extension. And there are a LOT of complied extensions in the
default channel, not to mention wheels ( which we could say "Don't do that"
about those)

I'm just not sure that features provide a robust solution to this problem.
But if they do, great.

One of the challenges is that if you mix run times, it generally won't fail
right away (or may not fail at all)

-CHB

[mcg1969]

I dropped out of this conversation, so I know I might either be repeating something already said or missing context---_but_, the more I think about it the more I believe that cleaning our our channel collision problem---that is, two different channels providing the same package---is going to solve a lot of problems, like this one.

Handling a version of Python compiled with a non-standard compiler is really no different than having to differentiate between 32-bit and 64-bit builds. Hence: each unique CPython needs to be in its own channel, and we need to make sure that those channels don't stomp on each other. Whatever channel is highest priority wins.

[mcg1969]

There are too many things being blocked by conda's inability to handle channel conflicts properly. Time to expedite that fix.

[ukoethe]

@mcg1969

Handling a version of Python compiled with a non-standard compiler is really no different than having to differentiate between 32-bit and 64-bit builds.

Interesting. Two questions:

• When your idea gets implemented, how will a user create an environment that uses a non-standard compiler?
• Apparently, you arrived at the conclusion that features are not the right solution to the compiler configuration problem. Could you explain why?

[mcg1969]

When your idea gets implemented, how will a user create an environment that uses a non-standard compiler?

Channels will need to be specifiable on a per-environment basis. That sounds like a user interface challenge (since after all you'll need to specify the channel list before you create the environment!)

Apparently, you arrived at the conclusion that features are not the right solution to the compiler configuration problem. Could you explain why?

Features are proving vexing to implement in a consistent manner with behavior that everyone is satisfied with.

For instance: what should take priority, the use of a feature, or a higher version number? For instance, suppose we have the mkl feature in an environment, and we run conda install numpackage. Further suppose that numpackage 1.0 uses the mkl feature, whereas numpackage 2.0 does not. What should be preferred? There is merit in both choices, but currently we choose the latter. And that's not likely to change since the mkl -> nomkl transition depends on it.

Furthermore, I don't think features alone address some of the mutual exclusion requirements that things like compiler versions will need. For example, if we use a feature to specify a particular compiler version, how do we prevent packages that were compiled with a _different_ compiler are not installed?

I don't want to say that I've definitively concluded that features aren't at least part of the solution. But if you had been in my shoes the last couple of months you wouldn't be as reluctant as I am to pronounce them the primary method of solution.

[mcg1969]

Continuing the numpackage example. Suppose we had two channels, openblas/ and mkl/, each populated with exactly the same packages, except as needed to link to their respective BLAS libraries. Perhaps numpackage 2.0 is only in openblas; an MKL version hasn't been built yet.

With the current regime, conda update numpackage is going to pull the OpenBLAS version, whether the user wants it or not. With two separate channels, and proper collision avoidance, an MKL user will never accidentally install the OpenBLAS version.

[ukoethe]

What should take priority, the use of a feature, or a higher version number? Currently we choose the latter

IMHO, the use of a feature should be preferred. If I'm not mistaken, that's what the new version resolution algorithm in conda 4.0 implements, no? (My preliminary tests of conda 4.0 seem to confirm this.) And it's also the effect of using different channels, if I interpret your openblas vs. mkl example correctly.

[mcg1969]

Everything I just described above regarding features is 4.0 --- including the problems. I'm simply not convinced they fully solve the problems you're hoping they will, and I think it will be far too easy to create environments with conflicting packages without _some_ further enhancements.

[ChrisBarker-NOAA]

I recall some talk about a way to specify platform-independent packages --
i.e. pure python packages don't need to be any different on Linux vs
Windows vs OS-X.

IF we could get that working, then Win-VC2105 could be a different
platform, and folks could still get pure python packages from the other
platforms.

Of course, they couldn't get compiled packages that have nothing to do with
python without re-building them....

This is a pain -- thank you MS!

-Chris

On Thu, Mar 31, 2016 at 12:27 PM, Michael C. Grant <[email protected]

wrote:

Everything I just described above regarding features is 4.0 --- including
the problems. I'm simply not convinced they fully solve the problems you're
hoping they will, and I think it will be far too easy to create
environments with conflicting packages without _some_ further
enhancements.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/conda-forge/conda-forge.github.io/issues/64#issuecomment-204089586

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

Chris.[email protected]

[msarahan]

We do have noarch builds, but they're sort of error-prone (https://github.com/ContinuumIO/anaconda-issues/issues/730) and not widely used.

[SylvainCorlay]

Sorry for jumping in so late.

I think that it would be a very bad idea to make a MSVC14 build of python for all the reasons mentioned earlier with respect to the ability to install binary wheels from pip that would then _not_ be binary compatible.

Besides, it is actually possible to build C extension modules with more recent versions for MSVC than the one used to build Python itself. The one thing that one must be cautious to use the MSVC8 C runtime when allocating or de-allocating objects passed or received from Python.

Actually, I am doing so (forcing the use of MSVC14 to build python 2.7 extensions) in the case of extension modules built with pybind11. Pybind11 (authored by @wjakob) has similar goals to boost.python but is C++11-only and therefore requires msvc14 on windows.

@jakirkham I made an example project here, built with pybind11, which comes with a conda recipe. The conda recipe's bld.bat does the magic of forcing MSVC14 for the build. You can also notice that I was careful to include the MSVC14 runtime as a runtime dependency to the conda package's meta.yml.

Finally, Python 3.5 is already build with MSVC14, which is excellent news because

1. MSVC14 already supports C++11 features
2. Starting with this version of MSVC, the C runtime was split into two parts: a compiler-independent part (the universal runtime) and the compiler dependent part. Future versions of MSVC will rely on the same universal runtime which will make it possible to build extension modules with future versions of MSVC without any concern about the C runtime version. (One more motivation to switch to Python 3.5 !)

I recommend this blog post by Steve Dower on the adoption of MSVC14 in Python 3.5, the split of the universal C runtime, and the consequences for authors of C extension modules.

[msarahan]

@SylvainCorlay thanks for the example. I think that's equivalent to setting the following in meta.yaml:

build:
  msvc_compiler: 14.0

This is not well documented, but went in a while ago: https://github.com/conda/conda-build/pull/600

As you mention, one must be careful about allocation and deallocation. How do you handle that in arbitrary 3rd party extensions? Do you use any special tools to find where those happen? Is it a reasonable task for people who are just building packages, rather than developing them? Can you point to code examples? Is it something that can be built into conda build, or does it need to be done on a per-recipe basis?

Another good article on this topic is http://siomsystems.com/mixing-visual-studio-versions/

[SylvainCorlay]

@SylvainCorlay thanks for the example. I think that's equivalent to setting the following in meta.yaml:

build:
  msvc_compiler: 14.0

Thank you for the pointer.

As you mention, one must be careful about allocation and deallocation. How do you handle that in arbitrary 3rd party extensions?

I don't think that it is possible.

Do you use any special tools to find where those happen?

No, but I imagine it is possible.

Is it a reasonable task for people who are just building packages, rather than developing them?

I think that it is up to the people developing the packages to be careful about it. If they use the right tools to write their extension modules rather than directly calling into the C Python APIs, it should be a automatic. The trick is to use PyObject_Malloc, PyMem_Free, PyObject_Delete etc. instead of free, malloc etc...

[ChrisBarker-NOAA]

On Thu, Mar 31, 2016 at 5:21 PM, Sylvain Corlay [email protected]
wrote:

I think that it is up to the people developing the packages to be careful

about it. If they use the right tools to write their extension modules
rather than directly calling into the C Python APIs, it should be a
automatic. The trick is to use PyObject_Malloc, PyMem_Free,
PyObject_Delete etc. instead of free, malloc etc...

I wonder if cython does this consistently? But in any case, this isn't
going to help for wrapping C libs not written specifically for Python....

Also, I thought there were other issues, like sharing objects from the C
lib -- file pointers for sure, but maybe others?

-CHB

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

Chris.[email protected]

[patricksnape]

Yes, I'm usually reluctant to discuss the intricacies of the the VS runtimes with package maintainers because it's really quite complicated to get everything working properly. For the average package maintainer, who probably works on Unix and then is just creating Windows builds for their users, this kind of careful runtime handling is probably out of scope. Although it is possible, and clearly @SylvainCorlay is an expert and a perfect example of the kind of person who _will_ be careful, I prefer to strongly suggest to people that this is the burden of Python 2.7. What better way to increase Python 3.5 adoption than to tell people that they _can't_ have Python 2.7 builds because they used C++11 :smile:? Also, it's been a while, but if I remember correctly, even with the new UCRT you don't actually get version independent binaries because any C code will still rely on vcruntime140.dll.

In terms of Cython, Cython prefers to delegate to PyObject_* commands, but the API is there to call libc directly in a fairly straightforward manner. So you can't really guarantee someone hasn't allocated using `malloc/free. I wonder what libcffi is like in this regard...

[ukoethe]

Package variants are a natural phenomenon -- users will always request conda to offer certain packages in several variants (for example, we are discussing vectorization instructions and BLAS elsewhere). Thus, conda needs powerful and reliable machinery to manage co-existing variants and to make sure environments remain consistent in their presence. Once such machinery is available, support for a non-standard compiler is simply Yet Another Variant and nothing to worry about. If we give the people concerned the right tools, they will take care of creating the variants they need, and make pull requests to sites like conda-forge so that everyone can profit.

Thus, I'd like to invite all you bright people to make constructive contributions to conda's variant management capabilities. Getting this right would really make a difference. Thank you.

[mcg1969]

I wonder if Conda's existing dependency resolution capabilities couldn't be hacked to at least _demonstrate_ proper class/variant behavior.

Suppose we create a metapackage called blas_variant; no contents, no dependencies. Instead of version _numbers_, use strings, one for each variant: openblas, mkl, etc. Whenever a package links to a specific BLAS, it includes as a dependency the appropriate version of the blas_variant metapackage. For example:

• MKL numpy uses depends: ["blas_variant mkl" ...]
• OpenBLAS numpy uses depends: ["blas_variant openblas"...]

Conda's natural dependency resolution will prevent multiple versions of blas_variant from being installed, and therefore ensure that all packages that use a BLAS in a given environment use the same one.

One case I'd like to handle is that OpenBLAS and MKL are installed in the same environment, but different programs use each. For instance, say Python uses OpenBLAS, and R uses MKL. This is handled using separate variant metapackages for each language; e.g. r_blas_variant and py_blas_variant. And it also points out that these variant dependencies should _not_ be added to the BLAS libraries themselves. Because they _can_ live alongside each other in theory, conda should not prevent them from doing so.

Setting aside the obvious aesthetic problems with this approach, does this capture the behavior we're looking for? What would its limitations be?

[mcg1969]

And while I _say_ there are obvious aesthetic problems with the approach, that doesn't mean we _shouldn't_ use metapackages if they provide the right behavior. What we _can_ do is enhance conda to properly _detect_ these types of metapackages and adjust its messaging so that the user better understands what they are and what they do. For instance, there is no such thing as an "upgrade" or "downgrade" in this case---everything variant change is a "crossgrade".

EDIT: In fact, perhaps a better way to go is to use build strings instead of version numbers; i.e., blas_variant 0 mkl to differentiate the variants. Obviously this looks less elegant, but again that's something conda can clean up.

[SylvainCorlay]

Although it is possible, and clearly @SylvainCorlay is an expert and a perfect example of the kind of person who will be careful, I prefer to strongly suggest to people that this is the burden of Python 2.7.

It is only that I was in exactly that situation of needing to build C++11 extension modules for both Python 2 and 3 on Windows.

What better way to increase Python 3.5 adoption than to tell people that they can't have Python 2.7 builds because they used C++11 :smile:?

I would agree with that.

[ukoethe]

@mcg1969

Instead of version numbers, use strings, one for each variant:

This is a cool idea, but I have to think about it in more detail before responding.

[ukoethe]

What better way to increase Python 3.5 adoption than to tell people that they can't have Python 2.7 builds because they used C++11?

When Python 3 fails to attract users by its capabilities, and we have to resort to political tricks like "you won't get C++ 11 if you don't upgrade", it's nothing to be proud of. Reminds me of Windows 8... :disappointed:

[patricksnape]

@ukoethe It starts to become a bit of a philosophical debate really! IMHO there isn't any reason to stick with 2.x outside of legacy code. Recompiling the entire Python ecosystem with a new compiler chain just to get C++11 and all of it's benefits seems like a lot more work to me than probably just wrapping some things in list and adding some parentheses to print! Plus, Python 3.5 _does_ have some really nice things for scientific computing like the @ operator!

[pelson]

I don't thing we want this thread to descend into a py2 vs py3 debate, so I'm going to steer the ship here :ship: :anchor: :smile:

I personally haven't seen enough benefit to invest my own cycles on supporting multiple MSVC versions for py27. The core Python devs have clearly made the decision not to make a py2 release that supports vc14 out of the box (though I accept they may have been influenced by politics and practicalities of not having a tool like conda at their disposal). For me, that is reason enough to avoid doing it if we can (its the same reason that most patches in recipes should have an upstream issue filed about them) - if we want to influence _how_ a piece of software builds, we should do that in the canonical source repository.

I completely accept though that if there are compelling reasons to do something, then we should consider doing it based on an assessment of benefits, risk, effort, user experience, and appetite for change.

[ukoethe]

I don't thing we want this thread to descend into a py2 vs py3 debate

I fully agree. That's why I'd prefer technical arguments over political ones like "What better way to increase Python 3.5 adoption ...".

[ChrisBarker-NOAA]

The core Python devs have clearly made the decision _not_ to make a py2
release that supports vc14 out of the box (though I accept they may have
been influenced by politics

Politics, maybe. If "we really din't want expend energy extending the life
of py2" is politics.

and practicalities of not having a tool like conda at their disposal)

I'm not sure it's that different -- if they (or anyone) releases a py2
binary for Windows built with a different runtime, it could be essentially
another platform, like 32bit and 64bit Windows is. And pip and wheel could
deal with it that way just fine.

But aside from not wanting to put energy into it -- they are also quite
concerned about the negative user experience if people get them confused.

I actually think we are in a very similar position with conda/conda forge,
except that at least we CAN build both python and a large number of
packages.

BTW, it might be worth checking out what stackless had done, I know that
wanted this a while back,

CHB

For me, that is reason enough to avoid doing it if we can (its the same
reason that most patches in recipes should have an upstream issue filed
about them) - if we want to influence _how_ a piece of software builds, we
should do that in the canonical source repository.

I completely accept though that if there are compelling reasons to do
something, then we should consider doing it based on an assessment of
benefits, risk, effort, user experience, and appetite for change.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/conda-forge/conda-forge.github.io/issues/64#issuecomment-204751552

[jakirkham]

Thanks @pelson for steering. :smile:

To clearly state my interests in this issue (and hopefully help keep us out of troubled waters), they are as follows.

1. See if there is community interest in non-standard Windows CPython builds.
2. See if a prototype can be developed by those interested.
3. Figure out where we go from their (patch CPython, go our own way, build system tweaks, conda changes, etc.)

On 1 the answer thus far seems to be some, but maybe not as much as expected (though we could simply just not be reaching the right people). On 2 the answer is yes for one case, but it probably needs to be worked on a bit more with some feedback. As far as 3, it seems we simply aren't there yet and it might be more clear once 2 is discussed more and completed.

[jakirkham]

After @ChrisBarker-NOAA comment, I found a related mail thread discussing Stackless Python and the need to have a different VS that is not 2008. It's a bit of a long read and covers some of the same ground we have already covered (e.g. the controversial Python 2.8, migrating to Python 3, other versioning options). One interesting suggestion in there, which may or may not be worth pursuing is having VS version hard-coded in the library names. This potentially would allow the same CPython distribution to be support with multiple VS versions, but allow for different compilers to be used. This also avoids the issues of pip not working due to an expected VC.

[jakirkham]

There is one important technical challenge to note. Namely the FILE* API. See refs below.

• https://mail.python.org/pipermail/python-dev/2013-November/130459.html
• https://mail.python.org/pipermail/python-dev/2013-November/130487.html
• http://bugs.python.org/issue15883

[jakirkham]

There were a few other issues (looks like 3) referenced in this post that a Microsoft developer confirmed were it.

[jakirkham]

Well, it looks like the conclusion to be drawn from this is best summarized in this post by the aforementioned Microsoft developer. Basically, the breakage would be too bad to consider using other VCs either for CPython 2.7 or extensions and that it should be prohibited. The remainder of the threader tapers off into stuff that we are not concerned with here before ending.

So, maybe the answer here is it would be technically unfeasible to approach this topic, unfortunately. Sorry to get people's hopes up here, but I don't think we are any better equipped to solve this problem. I am going to close this issue as won't fix. Though people are welcome to discuss further.

[pelson]

A healthy debate, and well conducted on all fronts. Thanks to everybody for keeping focussed on the issue and putting time & effort on making your points of view clear and succinct. :+1:

[jjhelmus]

Is a version of Python is built using a non-standard version of Visual Studio then then ABI tag should be changed so that wheels and other binaries build against a standad VC do not get installed. See PEP 425 for details.

[mwiebe]

I asked about this here: https://github.com/conda-forge/staged-recipes/pull/363#issuecomment-216706895, now found this issue.

Within the VFX industry, most of the major tools build Python 2.7 with a non-standard MSVC compiler, and talking to people using many of these tools together, our support of the standard CPython 2.7 in Deadline was actually the only one they had which worked this way. While it may not be recommended by Microsoft/CPython, Autodesk and other large vendors have been doing this for many years.

[ChrisBarker-NOAA]

THanks @mwiebe -- nice to hear of real field experience. But I'm confused about what "it" is:

"""While it may not be recommended by Microsoft/CPython, Autodesk and other large vendors have been doing this for many years.""

Do you mean building and distributing extensions to python 2.7 with MSVC 2010 or 2012, and users using them with the standard py2.7 built with VS 2008?

In which case, is it simply a matter of making sure that any extensions built this way don't share file handles with Python? or are there other gotchas -- and if so, are they documented anywhere? The file handle thing is the only one I know about, but I've heard tell that are arbitrary other corners of the lib where things can go wrong...

or is "it" in this case distributing python itself built with another version of VS? and then extensions to match?

[mwiebe]

The latter one, "it" in this case referred to building and embedding Python 2.7 with MSVC 2012 (or newer), and building extensions to match.

I think I overstated the Microsoft/CPython recommendation - the thing that is clearly discouraged is building extensions with a different compiler than the Python, but usually it's phrased as "build Python 2.7 plugins with MSVC 2008" because that's what the official CPython is built with. I've tended to break this rule too though, e.g. in DyND where we use C++14 features so require MSVC 2015, and building/linking that against Python 2.7 built with MSVC 2008 _seems_ to work fine, as we never transfer memory or handles ownership across the extension boundary.

[mwiebe]

Defining a particular ABI tag as @jjhelmus mentions for Python 2.7 with MSVC 2012, Python 2.7 with MSVC 2015, etc that anybody doing such builds could use would be really nice. I'd love to be able to have embedded Python in our applications done this way with conda preinstalled and able to install compatible extensions from conda-forge out of the box, then also have it interact cleanly with pip and other mechanisms via that ABI tag.

[ChrisBarker-NOAA]

This came up on the python-dev list a year or so ago (I think).

The Stackless folks proposed release=ing a 2.8 that would be the same
except it would be compiled with a newer MS compiler. THis was rejected
because it really makes NO sense to call something a different python
version just because it's a different compiler -- after all, python on
Linux, and Mac, and ... are all called Python 2.7 -- and they are not the
least binary compatible.

What I don't remember is why the ABI tag idea wasn't proposed (or wasn't
adopted) -- that would be easy and robust. But maybe it was that python.org
didn't want to host two builds of the same python version for Windows that
were incompatible.

The whole conda machinery does make all that easier -- though you would
still have the same problem -- when you selected a pyton 2.7 for Windows
you'd need to choose which one, and then there would be different binary
packages available depending on which one you'd choose -- kinda confusing
to the community in general, but maybe good for a "curated" system.

-CHB

On Wed, May 4, 2016 at 3:06 PM, Mark Wiebe [email protected] wrote:

Defining a particular ABI tag as @jjhelmus https://github.com/jjhelmus
mentions for Python 2.7 with MSVC 2012, Python 2.7 with MSVC 2015, etc that
anybody doing such builds could use would be really nice. I'd love to be
able to have embedded Python in our applications done this way with conda
preinstalled and able to install compatible extensions from conda-forge out
of the box, then also have it interact cleanly with pip and other
mechanisms via that ABI tag.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/conda-forge/conda-forge.github.io/issues/64#issuecomment-217018335

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

Chris.[email protected]