cvxpy dependency fails to compile - no wheels available

[skoudoro]

Hi @SteveDiamond and CVXPY developers,

Our Travis bots (DIPY) have different issues with cvxpy after your last release. We had no issues before.

As you can see here, building wheel for osqp is failing. We got this problem on many bots as you can see here

Any ideas?

[bstellato]

It looks like you do not have Python libraries and headers properly installed or that you have multiple python versions. CMake finds the executable but not the libraries and headers with the same version and returns an error.

Could you please try to specify the right variables that the installer needs? Here is an example with a very similar problem: https://stackoverflow.com/a/29657288/9753175

Here they describe how to set those variables in OpenCV.

Wheels are not available because pypi (pip) repositories do not allow it for Linux. If you want precompiled binaries for all the platforms I suggest you to go for anaconda.

[matthewgilbert]

I looked into doing this without conda but it seems rather involved for python > 3.4. The reason is .so files are not copied into the virtualenv (I think) as discuseed here and the default travisCI setup does not include the required python version.

You can see this from adding ls to the .travis.yml file.

$ python --version
Python 3.5.5

$ pip --version
pip 9.0.1 from /home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages (python 3.5)

$ ls -d /usr/include/python*
/usr/include/python2.7  /usr/include/python3.4  /usr/include/python3.4m

$ ls -d /usr/lib/python*
/usr/lib/python2.7  /usr/lib/python3  /usr/lib/python3.4

One possible work around would be to a new system wide install of python from an external PPA as discussed here and then set PYTHON_INCLUDE_DIR and PYTHON_LIBRARY accordingly but this seems extremely hacky since it depends on creating and using a system wide installation of python which is not recommended

The CI Environment uses separate virtualenv instances for each Python version. This means that as soon as you specify language: python in .travis.yml your tests will run inside a virtualenv (without you having to explicitly create it). System Python is not used and should not be relied on. If you need to install Python packages, do it via pip and not apt.

I think a better solution would be to move to using conda as discussed by @bstellato however I am currently experiencing issues getting the newest build via conda as discussed in #499.

To get this to work for 1.0.1 just make sure to add conda install -q -n test-environment -c conda-forge lapack

[skoudoro]

Thanks for your feedback @bstellato and @matthewgilbert !

Wheels are not available because pypi (pip) repositories do not allow it for Linux

??? I do not understand this affirmation. Our project has wheel for Linux, opencv or vtk has wheel for Linux as you can see here. What do you mean here @bstellato ?

but this seems extremely hacky since it depends on creating and using a system wide installation of python which is not recommended

I agree @matthewgilbert, I would like to avoid that

If you want precompiled binaries for all the platforms I suggest you to go for anaconda
I think a better solution would be to move to using conda

We want to make the life easier for our users so it should work with pip AND anaconda, there is no point to force the users to download and use anaconda on linux.... I think, having òsqp wheel on linux seems really important. How can we help for that @bstellato ?

can you have a look on this thread @arokem, @garyfallidis

[rileyjmurray]

@skoudoro

As I understand it: you cannot create python wheels for linux-based operating systems if a project has C dependencies (cvxpy has such dependencies). If you (or anyone on this thread!) thinks otherwise, please let me know. I'm currently working on syncing public deployments with Travis-CI builds, so now is a good time to resolve questions of wheels-vs-eggs-vs-source.

[arokem]

I believe that is what the manylinux project is supposed to enable:

https://github.com/pypa/manylinux

As you can see in the links that @skoudoro provided above, projects such as VTK and OpenCV have adopted this as their practice. It would help downstream dependencies (such as our project) if these wheels could be routinely provided for cvxpy as well.

Thanks for the great project!

[Garyfallidis]

Hi @SteveDiamond, @rileyjmurray , @bstellato and all,

I am not sure I understand here. Are you planning to support a stable pip install for cvxpy or only a stable conda installer? Currently in your travis setup it seems you are only testing with anaconda. As cvxpy is a core module for optimization my suggestion is to provide a stable pip version too. This would definitely be useful for DIPY for example https://github.com/nipy/dipy I am sure there are many other projects with similar needs. It would help immensely if you had another travis bot running with pip and not only conda.

[bstellato]

@skoudoro @Garyfallidis regarding OSQP we are always automatically building and deploying both anaconda and pypi versions of the solver. See osqp-python.

As @rileyjmurray said, it is not currently possible to upload pypi binary wheels that include C extension binaries. That's why in the automatic OSQP deployment we build a sdist wheel instad for linux. Unless something changed in the last pypi policies, the twine upload command would return an error if you are uploading a linux bdist wheel.

@arokem I would be very happy to provide bdist wheels for OSQP using manylinux. However I have no experience with that. Is there anyone who could help with it?

[rileyjmurray]

@Garyfallidis the Travis CI setup installs cvxpy from source (a la python setup.py install after cloning the cvxpy repo). The only reason for using anaconda is to get all the dependencies in order. I agree with the importance of having cvxpy readily available on pip. However it is not clear to me how using pip in continuous integration builds would be helpful.

The issue you are describing suggests that cvxpy would benefit from tests that install with calls such as pip install cvxpy or conda install cvxpy. The thing is- I'm not sure of a service that expects this kind of usage. Do you have advice on running these tests in an automated way?

[Garyfallidis]

Hi @rileyjmurray, a few clarifications. We tested the pip installer for cvxpy in a linux machine and it works with no issues. The problem is only with our travis bot. @skoudoro will test to see if adding the include and lib dirs helps the issue.

Remember that the issue that we have in DIPY's travis bot is "Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS) (Required is at least version "3.5")"

As about your question. You can think of continuous integration as a way to check the different tests in your code and make sure everything is passing. But also you can think of ci as a way of checking that your different installers are working in different os versions and operating systems and especially when things are not in order. It seems to me that it would be beneficial to have a travis bot running with a pypi installer in a standard linux environment without anaconda and another one which will be using anaconda. Right now your travis bot is not building at all, only Appveyor is building. You should expect that you still have many users that will run cvxpy in Linux and many of them will not have anaconda installed. Although it is true that more and more people are switching to conda, we still need to support the rest of the cases.

So, let's try to share some experience with the different ci systems used in cvxpy and DIPY. Will get back to you asap about this. We also need to have an anaconda travis bot which we haven't done yet. But let's figure out this issue first.

[bstellato]

@Garyfallidis I am trying to build the manylinux wheels for OSQP and it does not seem to be that easy. See https://github.com/oxfordcontrol/osqp-python/pull/5 for more details.

[matthewgilbert]

But also you can think of ci as a way of checking that your different installers are working in different os versions and operating systems and especially when things are not in order. It seems to me that it would be beneficial to have a travis bot running with a pypi installer in a standard linux environment without anaconda and another one which will be using anaconda.

@Garyfallidis Something to keep in mind which I mentioned above is that for TravisCI, the python environments are managed using virtualenv which does not create new shared object files. Also, the TravisCI Ubuntu Trusty machine only comes with 2.7 and 3.4.

I agree testing pip installing using some appropriately provisioned box with TravisCI would be good, but this doesn't address the issue where downstream users are trying to use virtualenv on TravisCI to test their packages which depend on cvxpy. This seems like a fundamental issue due to virtualenv not creating share objects and TravisCI only having system installs of 2.7 and 3.4, with no clean solution other than using conda. As others have mentioned above it seems like providing a wheels distribution of cvxpy could also address this?

[arokem]

Yes - I believe that building the manylinux wheel would be helpful for
this! I don't fully understand the issues you face in
oxfordcontrol/osqp-python#
https://github.com/oxfordcontrol/osqp-python/pull/55, but I will try to
take a look.

On Fri, Jun 1, 2018 at 12:57 AM, Matthew Gilbert [e-mail]
wrote:

But also you can think of ci as a way of checking that your different
installers are working in different os versions and operating systems and
especially when things are not in order. It seems to me that it would be
beneficial to have a travis bot running with a pypi installer in a standard
linux environment without anaconda and another one which will be using
anaconda.

@Garyfallidis https://github.com/Garyfallidis Something to keep in mind
which I mentioned above is that for TravisCI, the python environments are
managed using virtualenv which does not create new shared object files.
Also, the TravisCI Ubuntu Trusty machine only comes with 2.7 and 3.4.

I agree testing pip installing using some appropriately provisioned box
with TravisCI would be good, but this doesn't address the issue where
downstream users are trying to use virtualenv on TravisCI to test their
packages which depend on cvxpy. This seems like a fundamental issue due
to virtualenv not creating share objects and TravisCI only having system
installs of 2.7 and 3.4, with no clean solution other than using conda.
As others have mentioned above it seems like providing a wheels
distribution of cvxpy could also address this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cvxgrp/cvxpy/issues/503#issuecomment-393708547, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAHPNqkrm9ixUknBSKfR_kbmo_Gh4t8oks5t4HVogaJpZM4UOpJI
.

[matthew-brett]

Hi - @arokem pointed me here, because I build a lot of Linux wheels.

I just wanted to check that y'all know that:

• Manylinux wheels always have C extensions, of which Numpy, Scipy and many others, are examples. Manylinux has a related utility, auditwheel, that finds and vendors all necessary C libraries inside the wheel, as a post-processing step.
• There are a lot of conda users, but there are also a lot of pip users. I don't know the relevant proportions, but I would venture a guess that the proportion of pip users is not small. I don't think it's possible to guess yet, whether one will dominate at some point in the future. As pip distribution improves, it's possible that conda will recede, it's possible it will work the other way.
• CMake can be a problem building Python extensions, because it typically insists on finding a Python library to link to, and, for portable wheels, that's the wrong thing to do. In particular, on Debian-based distributions, there is frequently no libpython library installed. See: https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1 for a discussion. distutils builds, I believe, set compiler flags to allow Python symbols to be resolved at run-time, making the Python library unnecessary (because the extension is running within the Python binary process, where all the relevant symbols are loaded).

See https://blog.tim-smith.us/2015/09/python-extension-modules-os-x/ for more detail, and https://github.com/pypa/manylinux/issues/69 for a related issue.

I'd strongly encourage you to use multibuild (warning, I'm the maintainer for that project). I encourage you because it takes care of some of the machinery you need, and it's widely used (by Numpy, Scipy, Matplotlib etc), therefore, relatively well supported.

[SteveDiamond]

This is all quite helpful! I personally find working with the Travis environment extremely annoying, which is why I moved to the simplest possible way of providing dependencies. I can see the merit in having a separate Travis CI test that uses pip to provide the dependencies, but this is not a small feat in my view.

It's most important to have one method of installation that works perfectly, with no complexity to the process. In general people who use pip are more computer savvy, which is why I prioritized making the conda installation path simple. I also formed a bad impression of pip from when I first started the project, which may be erroneous now. In any case, the installation path with pip should be as easy and well tested as possible, but we all are busy people so we need to prioritize.

[matthew-brett]

It's really not too hard to set up the dependencies. I suppose you mean things like C libraries. Have a look, for example, at the Python Pillow recipe - it just downloads and builds all the libraries it needs in the Manylinux container, then builds the wheel, where the build process detects the libraries automatically. Finally, (automated by multibuild), auditwheel packs all the required C libaries into the wheel, and creates a Manylinux wheel.

I think you'll find that, after a couple of hours of initial investment, you'll have wheel builds automated with a very small ongoing maintenance cost. I'm very happy to help with multibuild.

I don't think pip users are more computer savvy - pip is the method of installation recommended by the Python Packaging Authority, so many users just assume that it's going to work for them. Luckily, at the moment, it very often does, but that certainly wasn't true a few years ago, before the widespread distribution of macOS, manylinux and Windows wheels.

I'm going to guess that about half your _potential_ users are using pip. What will likely happen is they will try and install via pip, it will fail some horrible way that they don't understand, and then they will give up, and you won't hear from them.

[Cerebrock]

matthew is right... walking down that path right now

[rileyjmurray]

@Cerebrock , can you be more specific? What is "that path"?

[demba]

I'm trying to install cvxpy inside a virtualenv on Ubuntu 16.04 and am getting an error similar to the one that started this thread (see below). I want to check with you guys if the error is related to this thread so I can monitor it and not file a separate ticket. Tks.

-- Could NOT find PythonInterp: Found unsuitable version "1.4", but required is at least "3.6" (found /home/me/.pyenv/shims/python3.6)
-- Could NOT find PythonLibs (missing:  PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS) (Required is at least version "3.6")
CMake Error at CMakeLists.txt:196 (message):
  You need Python libraries to build the Python interface

[bstellato]

Yes it is. It is a problem related to OSQP. I will try to fix it in the next days.

[Garyfallidis]

Thank you @bstellato that would be fantastic! 👍

[bstellato]

New manylinux wheels available on pypi for osqp! https://pypi.org/project/osqp/ Let me know if it fixes the issue.

[Garyfallidis]

Congratulations @bstellato! The DIPY developers salute you! We are currently testing your new wheels!!! :tada:

[skoudoro]

Awesome ! Thanks a lot @bstellato! I just checked it and everything works like a charm!

Thanks again! I think I can close this issue

[demba]

@bstellato, Tks much! Was able to successfully pip install cvxpy in a virtualenv.

Demba.

[handrew]

I am also having the same problem mentioned at the beginning of this thread. :( I'm on Python3.7.

What was the resolution here?

    -- Could NOT find PythonLibs (missing:  PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS) (Required is at least version "3.7")
    CMake Error at CMakeLists.txt:196 (message):
      You need Python libraries to build the Python interface


    -- Configuring incomplete, errors occurred!
    See also "/private/var/folders/1k/1prtczss35v7y8kjrkp9dzpw0000gn/T/pip-install-bmtencom/osqp/osqp_sources/build/CMakeFiles/CMakeOutput.log".
    make: *** No rule to make target `osqpstatic'.  Stop.
    error: [Errno 2] No such file or directory: 'osqp_sources/build/out/libosqpstatic.a'

[bstellato]

We do not have Python wheels available yet for OSQP in Python 3.7. Many packages haven't been updated to Python 3.7 yet. Can you try with Python 3.6? We will make a new OSQP release soon and it should have Python 3.7 support.

[handrew]

Yeah I'm not tied to 3.7, thanks!

[wkschwartz]

We are tied to 3.7, so we're looking forward to those OSQP wheels!

Is there anything I can do to work around the problem at the top of this thread in the mean time? We're a pip shop, so switching to conda would be too costly.

[rileyjmurray]

@bstellato can you add a source distribution of OSQP to pypi? Right now you only have wheels.

Although source distributions wouldn't help Windows users much (in that they would need appropriate compilers installed), they would take care of mac and linux users who are in situations like @wkschwartz.

[wkschwartz]

cvxpy installs just fine locally on my Mac with pip. Having trouble getting it to work on Travis.

[bstellato]

I have just added the source distribution. I will upload the wheels for Python 3.7 very shortly. I am right now waiting for Travis and Appveyor to do their job. They can be very slow when you support many platforms and python versions.

[bstellato]

Wheels for python 3.7 for mac and linux are available. Windows and python 3.7 still have lots of issues with numpy and scipy and we are not able to build the wheels yet. @wkschwartz Please let me know if the installation works now.

[wkschwartz]

Everything works!

Thank you for the help and thank you for doing it so quickly!