Mu: FYI: the Mu wheel ignores the Raspberry Pi check

[bennuttall]

The wheel uploaded to PyPI was built on an x86 machine so the platform check obviously did not exclude packages not available for Raspberry Pi: https://github.com/mu-editor/mu/blob/master/setup.py#L25:L29

Wheels do not bundle setup.py - this is done at the point of building the wheel, so someone pip installing Mu on a Pi would get the wheel which didn't exclude those packages, and would fail (even if #669 didn't apply).

There's no good solution to this, as making the wheels platform-specific would mean providing one for every non-Pi platform. I have a couple of suggestions but really whether anything changes or not, the process will need documenting so people don't get tripped up by it.

One solution could be that I build a Pi platform wheel and host it on piwheels. Mu developer docs would have to suggest "To pip install on a Pi, run sudo pip3 install mu-editor -i https://www.piwheels.org/simple".

Another is you tell people to build from source when on a Pi, rather than using the wheel. That would be sudo pip3 install mu-editor --no-binary mu-editor.

Anyway, I just wanted to highlight this as an issue for anyone trying to pip install Mu on a Pi. Most users will be using apt, but if anyone wants an unreleased version or wants to develop for it, it will be handy to know what's going on.

[ntoll]

@bennuttall thanks for this... great stuff!

How does this relate to the development install instructions in the dev docs here:

https://mu.readthedocs.io/en/latest/setup.html#raspberry-pi

???

[illume]

I think environment markers are used for this these days.

An example from PEP 426: https://www.python.org/dev/peps/pep-0426/#dependencies

"dependencies":
  {
    "requires": ["SciPy", "PasteDeploy", "zope.interface > 3.5.0"]
  },
  {
    "requires": ["pywin32 > 1.0"],
    "environment": "sys_platform == 'win32'"
  },
  {
    "requires": ["SoftCushions"],
    "extra": "warmup"
  }
]

ED: this was wrong. See post below.

[carlosperate]

While using the environment markers sounds like a good change, these are still included in the setup.py file, isn't?

If that's the case, based on @bennuttall post it wouldn't affect the built wheels unless we build for/from different platforms, so it sounds like that would still cause the same issue with the uploaded wheels.

If a wheel is built specifically for the ARM platform I think we should ensure all the dependencies are available via PyPI. Otherwise is not of much use if it fails to install because of missing dependencies, or if it gives the user the wrong impression that it has been successfully installed, but then Mu fails to launch because of missing dependencies that can only be installed via the OS package manager.

There's no good solution to this, as making the wheels platform-specific would mean providing one for every non-Pi platform.

I thought there was a "armv6" or "armv7" platform that should work on most SBC?

One solution could be that I build a Pi platform wheel and host it on piwheels. Mu developer docs would have to suggest "To pip install on a Pi, run sudo pip3 install mu-editor -i https://www.piwheels.org/simple".

This sounds like a good option, but I believe it can be quite difficult with the current state of the PyQt5 wheels for ARM. I assume we are still a bit far away from having these wheels available in piwheels?

Another is you tell people to build from source when on a Pi, rather than using the wheel. That would be sudo pip3 install mu-editor --no-binary mu-editor.

If the source is included in PyPI, and the wheels available are marked for all known-working platforms (so they would exclud for ARM), would "sudo pip install mu-editor" on a Raspberry PI not default at to install from source?
That could probably achieve the same result without using additional flags, no?

[illume]

Hi ya,

Environment markers are for exactly this problem, and work in wheels. There have been slight modifications, but the feature goes back to 2005 or so and even earlier (pre .egg even).
There's better docs for them here: https://www.python.org/dev/peps/pep-0508/#environment-markers

I made a fork of the sampleproject package from pypa which shows how it can be used.
https://github.com/illume/sampleproject

If you want, you can confirm for yourself there that it works still (like I did). Make a wheel (python setup.py bdist_wheel), then install the wheel. Note the different dependencies.

Here's the part I changed:

    install_requires=[
        'peppercorn',
        'pygame; sys.platform=="linux2"',
        'numpy; sys.platform=="darwin"',
    ],

Note: there is support for them in pip. See their documentation here: https://pip.pypa.io/en/stable/reference/pip_install/#using-environment-variables

[carlosperate]

Thanks @illume, just had a play with your sample project fork and I see what you mean now.
All the platform specific markers are stored in the wheel, and the wheel itself is marked as none-any platform. So, each platform would install the dependencies indicated by the markers.

The problem we currently have with our setup.py is that the requirements included in the setup() function are arguments processed while the wheel is being created, and you end up with a different wheel depending on the platform used to build it.

Environment markers is definitely the way forward to included different dependencies per platform and also ensuring that pip install - e . works.

There is still the question of "should a wheel install successfully if you don't have all packages installed"?
We currently exclude the PyQt packages in setup.py for the ARM platforms to be able to use pip install -e ., and we depend on Rasbian to have the dependencies installed via apt (which is also why the virtualenv is created with the --system-site-packages flag).

If we create wheels that don't install PyQt5 in ARM platforms, then pip install mu-editor will install successfully, but the user will not be able to launch it unless they happen to have these packages installed via their package manager:
https://github.com/mu-editor/mu/blob/f89fc697ca67df9f71447b7f75aae521e4882fff/setup.py#L27

There is no supported way to add output messages to pip install but perhaps on __main__.py we can try to import these and if it fails in the ARM platform it could show a useful message about how to install them, or a link to the right instructions (we could even open their default browser to that link). It's hacky but can't think of a better way, other ideas are most welcome!

[illume]

Yeah, a bit of specific code at import time sounds the way to go.

(or... hacko way sudo apt-get install -y ... ;)

[carlosperate]

@ntoll I've tested and submitted #680 which reimplements the current setup.py with environmental markers (thanks @illume!) and merges the requirements.txt and requirements_pi.txt files together using markers as well.

Next step would be to add a bit of code at import time to detect missing PyQt5 dependencies when running on ARM and point the user to https://mu.readthedocs.io/en/latest/setup.html#raspberry-pi and possibly https://codewith.mu/en/howto/install_raspberry_pi (although if a user has purposely tried to install via PyPi in their Pi, then perhaps the first link is more useful? Not sure).
Unless we can think of a better solution?

[tmontes]

All,

I've just confirmed that installing Mu based off a wheel works nicely on the Raspberry Pi, running "Raspbian GNU/Linux 9.9 (stretch)".

Here's what I did:

• Created a wheel from the current master with python setup.py bdist_wheel. This was actually done on macOS, but could have been completed on any supported platform, given that the wheel itself is Python only. This produced the mu_editor-1.1.0a1-py3-none-any.whl wheel file.
• On Raspbian:
  
  
  
  • Ensured the Raspbian specific packages -- python3-pyqt5, python3-pyqt5.qsci, and friends, as documented here, were installed.
  
  
  • Created a virtual environment with python3 -m venv --system-site-packages venv.
  
  
  • Activated the virtual environment.
  
  
  • Copied the wheel file to Raspbian.
  
  
  • Installed with python -m pip install mu_editor-1.1.0a1-py3-none-any.whl.
  
  
  • Launched Mu with the mu-editor command.
  
  

I did not test any further, but I have no reason to believe this is any different from installing and running Mu from within a development environment. Everything should work, thus. :)

Given this, I wonder if this issue can be considered "fixed" and thus closed, or if there still is any pending and related concern. Thanks in advance for feeding the issue back.

[carlosperate]

I think we still have the issue that pip3 install mu-editor in Raspberry Pi installs successfully but is broken if the PyQt dependencies hasn't been installed via apt-get, and the virtualenv configured to uses system packages.
If I remember correctly there was a suggested "second step" to perhaps check for the imports and display a useful error message if not found, but we never got any feedback about that.

[tmontes]

If I remember correctly there was a suggested "second step" to perhaps check for the imports and display a useful error message if not found, but we never got any feedback about that.

Understood. There's no obvious solution at hand, I'd say. Here's my perspective:

• Raspbian users will, ideally, install Mu from the Raspbian APT repository as documented here: https://codewith.mu/en/howto/1.0/install_raspberry_pi
• The only motive to go pip install mu-editor is to grab the latest PyPI published version because the Raspbian APT repository may be out of date. How much would that be I have no idea, and which mechanisms are in place to ensure that once a new PyPI release is published the Raspbian APT repository is updated, again, are things I know nothing about.

With that in mind, I'd say we're entering "advanced beginners" territory: using the CLI in install Mu.
Facts:

• Even though https://www.piwheels.org/ says it carries distributions like PyQt5 and QScintilla, in practise it does not -- apparently the respective "detail pages" indicate several failed builds.
• That's why, I suppose, environment markers in the distribution omit them from the installation runtime requirements.

More:

• I know that the underlying issue -- specifying native package install/library dependencies (IOW: something that pip cannot and should not touch) -- has been brought to the PyPA's attention, but that's not an easy problem to tackle and no solution is in sight, AFAICT.
• Thus, the only possible solution I can foresee is, like you mentioned, catching import failures and printing out a helpful message -- yes to STDOUT, give that we're at the CLI, in advanced beginner territory.

The basic idea would be converting what currently is in mu/__init__.py:

from PyQt5.QtCore import QLocale                                                          

...to:

try:
    from PyQt5.QtCore import QLocale                                                          
except ImportError:
    print('simple explanation message including URL with detailed instructions')

Questions:

• Maybe the print could be conditionally executed based on sys.platform == 'arm'?...
• If so, what would the else clause do?
• Would we want to pollute the whole codebase with such try/except statements around native Raspbian dependencies?... That feels annoying and brittle!

More:

• Maybe the URL could be a variation of https://codewith.mu/en/howto/1.0/install_raspberry_pi: a properly named "advanced version" at the same website? (keep in mind, we're not talking about setting up a dev environment).

Not sure, but happy to hear ideas. :)

[bennuttall]

piwheels will soon calculate and show apt dependencies on the project pages, so that will help.