Mu: Windows installer dependencies in other OS

[carlosperate]

Continuing a parallel conversation that originated in: https://github.com/mu-editor/mu/issues/340 - Windows installer - what python libraries to 'package'

@carlosperate wrote in https://github.com/mu-editor/mu/issues/340#issuecomment-366994893:

Out of curiosity, what's the plan for these dependencies with the deb installer for Raspbian? Are they all going to be installed from apt packages in the global python3 environment?

@ntoll wrote in https://github.com/mu-editor/mu/issues/340#issuecomment-366995291:

@carlosperate that would be the most sensible thing to do AFAICT.

[carlosperate]

Having all that installed with Mu is probably fine for Raspbian, but maybe it's not all that great if we are advertising the deb file as an installation mechanism for other debian based distros.
If I was installing Mu in Ubuntu and it used the global python3 environemnt, and I'd probably be a bit annoyed if it also installed all of those packages.

Perhaps the deb packages could be separated in two?
1) mu-editor-only: Pretty much what we currently have, the basics to get Mu running.
2) mu-editor: To include mu-editor-only and all the python packages?

[ntoll]

The only packages that are additional to those needed by Mu to function properly given all the available modes (PyGame, Jupyter REPL etc) are:

• GPIOZero
• PiGPIO
• GuiZero
• Pillow
• Requests

[carlosperate]

Fair enough, they are not that many so far, but we are getting to a point where we are really polluting the site-packages with Mu dependencies. Installed in a new virtualenv we end up with:

certifi==2018.1.18
chardet==3.0.4
colorama==0.3.9
cycler==0.10.0
decorator==4.2.1
gpiozero==1.4.0
guizero==0.4.4
idna==2.6
ipykernel==4.8.2
ipython==6.2.1
ipython-genutils==0.2.0
jedi==0.11.1
jupyter-client==5.2.2
jupyter-core==4.4.0
matplotlib==2.0.2
mu-editor==1.0.0b14
numpy==1.14.0
parso==0.1.1
pgzero==1.1
pickleshare==0.7.4
Pillow==5.0.0
prompt-toolkit==1.0.15
pycodestyle==2.3.1
pyflakes==1.6.0
pygame==1.9.3
Pygments==2.2.0
pyparsing==2.2.0
PyQt5==5.9
pyserial==3.4
python-dateutil==2.6.1
pytz==2018.3
pyzmq==17.0.0
QScintilla==2.10.1
qtconsole==4.3.1
requests==2.18.4
simplegeneric==0.8.1
sip==4.19.7
six==1.11.0
tornado==4.5.3
traitlets==4.3.2
urllib3==1.22
wcwidth==0.1.7

Is PiGPIO a Raspberry Pi only module? It includes is a C library and daemon for GPIO control, doesn't it? so what would happen if it's installed on an Ubuntu PC?

Also TK will become a needed dependency for the deb in some linux distributions.

[ntoll]

paging @bennuttall (see @carlosperate's question)

[bennuttall]

pigpio is a C library made for doing physical GPIO stuff. It's included in Raspbian but won't need to be bundled with Mu.

python-pigpio is a Python library which acts as a client to a pigpio daemon running on a Pi. The client itself can be any PC. I have it installed on my Ubuntu laptop, you can install it on Windows and Mac - it's pure Python and is merely the implementation of socket communication with a remote Pi running pigpiod. The project's github repo bundles the C lib and Python lib together.

python-pigpio is published on PyPI as pigpio and in the Raspbian apt repo as python-pigpio/python3-pigpio. It's also published on the Raspberry Pi Desktop x86 apt repo, and Raspberry Pi's PPA.

Hope that helps.

[carlosperate]

Thanks Ben, that was very helpful.

I assume the python3-pigpio deb package is not available in the Ubuntu repos?

If not we could probably set it up as an "optional" dependency (would that be the "suggest" relationship? Not sure: https://www.debian.org/doc/debian-policy/#binary-dependencies-depends-recommends-suggests-enhances-pre-depends).
If the concept of optional dependency does not exists, a last resort could be to add a dummy package as an alternative (https://www.debian.org/doc/debian-policy/#syntax-of-relationship-fields).

[waveform80]

Fair enough, they are not that many so far, but we are getting to a point where we are really polluting the site-packages with Mu dependencies. Installed in a new virtualenv we end up with:

If you really wanted to split the deb the traditional naming would be something like mu-editor and mu-editor-extra but frankly I wouldn't worry about it. The list of (indirect) deps is quite long but also pretty typical for a top-level application. To see how horrific it can get, install debtree and have a look at some Python based applications:

$ sudo apt install debtree graphviz imagemagick
$ debtree veusz | dot -Tsvg | display svg:-
... huge graph ...
$ debtree --no-recommends meld | dot -Tsvg | display svg:-
... huge graph ...
$ debtree --no-recommends guake | dot -Tsvg | display svg:-
... huge graph ...
$ debtree guake | dot -Tsvg | display svg:-
... segfault! ...

However, unlike a venv these deps will get shared by anything else on the system that needs them which, looking at the list, is pretty likely in many cases (how much stuff depends on pytz, numpy, six, requests, etc). Basically: don't worry :)

If not we could probably set it up as an "optional" dependency (would that be the "suggest" relationship? Not sure: https://www.debian.org/doc/debian-policy/#binary-dependencies-depends-recommends-suggests-enhances-pre-depends).

Yup, depends is "mandatory dependency", recommends is "optional but apt will automatically install unless explicitly told not to (--no-install-recommends)", and suggests is "optional but apt won't automatically install unless explicitly told to (--install-suggests)" (or at least that's how I tend to think of them when packaging stuff).

[bennuttall]

I assume the python3-pigpio deb package is not available in the Ubuntu repos?

Not the official Ubuntu repo, but it is available for Ubuntu in our PPA - as would Mu. So the easiest first step to support installing Mu in Ubuntu would be to add it to our PPA, then the instructions would just be:

$ sudo add-apt-repository ppa:rpi-distro/ppa
$ sudo apt update
$ sudo apt install mu

then once we're at that point, we can see about getting ab Ubuntu (or Debian) sponsor to upstream it.

[bennuttall]

@XECDesign right?

[XECDesign]

Afraid I can't comment on getting the package upstream as I haven't been through that proccess. No problem adding Mu to the PPA though.

[carlosperate]

Thanks for the clarification @waveform80, sounds like we can have the other dependencies as recommend and so users that do not want the extra packages can avoid them with the right flag (just gotta make sure we document it :).

@bennuttall @XECDesign If Mu is added to the rpi PPA I think that's more than enough for our userbase. I was picturing a situation where users would have to download a deb file and need all the dependencies available in the main Ubuntu repo, but this way is much better and makes a lot more sense to do it that way, sounds great!

[ntoll]

FYI... I'm about to start the process of getting Mu into Debian, which is upstream of all the things (we're already in Fedora, the other "upstream of everything").

As I understand it, it should be relatively "easy" (note the quotes) to get Travis to build our .debs and .rpms as part of the CI process.

[carlosperate]

Cool, having it upstream would be great! but keep in mind that we will need to make sure all other dependencies are upstream too, like GPIOZero, PiGPIO and GuiZero.

[bennuttall]

No problem adding Mu to the PPA though.

Yes @XECDesign that's all I meant :+1:

@bennuttall @XECDesign If Mu is added to the rpi PPA I think that's more than enough for our userbase

Exactly. That will be a great position to be in.

FYI... I'm about to start the process of getting Mu into Debian, which is upstream of all the things (we're already in Fedora, the other "upstream of everything").

Excellent!

As I understand it, it should be relatively "easy" (note the quotes) to get Travis to build our .debs and .rpms as part of the CI process.

FYI Launchpad automatically builds .debs for multiple Ubuntu versions for the PPA :+1:

keep in mind that we will need to make sure all other dependencies are upstream too, like GPIOZero, PiGPIO and GuiZero.

Yep

[ntoll]

OK... so it sounds like we need to get organised to ensure "all the things" are upstream (if that makes sense).

[waveform80]

Actually ... I'm not sure you do have to upstream "everything". Remember that "Recommends" isn't a hard dependency: it's entirely possible to install something without its recommendations too. So, the question becomes "can a package recommend other packages which don't exist?" (i.e. could a hypothetical upstream mu-editor package "recommend" gpiozero if gpiozero wasn't also an upstream package - what would apt do at installation time? Quit with an error? Proceed with a warning?).

I just ran a little experiment on my Pi here: re-built the latest gpiozero package with "Recommends: quux" in the control meta-data (there's no quux package in Raspbian). Interestingly, lintian (the debian package checker) doesn't complain about it so I'm vaguely hopeful it's going to play nice...

... and it does! Just tried installing my package with "Recommends: quux" and apt lists the recommendation in its output, but doesn't complain about it being missing, and simply proceeds with the installation as normal.

So: I don't think pigpio, gpiozero, etc. have to be upstream, they just have to be listed as optional dependencies. Of course, Debian policy might have something to say about deliberately missing "Recommends" ... probably best to ask an official maintainer.

[bennuttall]

As long as the editor works without them, that's no problem. For example, without pygame/zero, the pygame-zero mode wouldn't be available, but would other available modes would work ok.

Slightly off-topic: perhaps a "modes manager" would be an interesting addition?

[tjguk]

@bennuttall in fact, per #343, Pygame Zero mode would be available but would silently fail to run if Pygame/Zero weren't installed. But that will be fixed.

[ntoll]

@tjguk thanks for the reminder. I fixed that in the most recent beta release. See: https://github.com/mu-editor/mu/blob/master/mu/app.py#L91 :-)

[carlosperate]

Okay, so is this conclusion correct?

• For the Raspbian/debian Mu package, all Python dependencies will be installed via deb pacakges
• The packages required for Mu to run would be "mandatory" and are already available in the upstream debian repos
• The packages that will be listed as "Recommends" are for the additional Mu modes, and some of them are not available in upstream debian (but that's fine). These are:
  
  
  
  • GPIOZero
  
  
  • PiGPIO
  
  
  • GuiZero (I think this one might need a debian package for the Raspbian repo first)
  
  
  • Pillow (Is this one available as a deb in Raspbian?)
  
  
  • requests
  
  
• To start, Mu will be added to the Raspbian PPA and that will offer all the deb files (but it means users have to manually add the PPA, which is just an extra command before apt-get install).
• The Mu package could then be submitted upstream, where not all the "Recommends" packages are available

Does that cover everything?

[bennuttall]

It doesn't make sense to me that those packages are recommended (or required if you're installing from pypi).

• Qt itself is obviously a hard dependency, as Mu couldn't run without it.
• QtCharts could be "recommended" but not essential, as Mu can run without it but it unlocks a Mu feature (the plotter). The same could be said for pygamezero (only on that Mu contains a mode for it, but I think that's a weak connection).
• gpiozero/guizero/etc are really unrelated to Mu, they're just libraries the user might choose to use. Mu doesn't have any features specifically for them. Of course I'm in favour of people using them but it just seems odd to formalise this in this way, they're completely unrelated to being dependencies.

Pillow (Is this one available as a deb in Raspbian?)

Yes, and pre-installed as it's a requirement of sense-hat.

[waveform80]

Pillow is python3-pil in Debian & Raspbian (used to be python3-imaging but that's now a virtual which redirects to python3-pil). You're correct that GuiZero is not currently deb-packaged.

Can't add anything else beyond what Ben's posted above.

[ntoll]

OK... so Debian/Raspbian packaging makes me feel like I'm trapped in a maze of twisty dependencies armed only with my vorpal blade. Unfortunately "Xyzzy" doesn't work for me.

The end goal is for any user of Raspbian never to have to install Mu. It's already there along with the dependencies to enable its full feature set (Plotter, PGZero, GPIOZero etc). This is likely to have to include @carlosperate's wonderful work on compiling QCharts for ARM.

Off the back of this, we will have some .deb packages we can push upstream into the wider Debian eco-system, hopefully in whatever is current testing. In this case, I believe that when I apt-get install mu-editor I'd want to see all the features mentioned in the Mu documentation and that will be at the end of the URL you get sent to when you click the "Help" button.

If we can package the currently unavailable modules (Mu itself, QCharts, PGZero, GPIOZero etc) and they are accepted into Debian testing, I'm left wondering why they'd just be recommended dependencies.

In any case, I suggest the first stage is getting Mu into a complete, packaged and installable form for Raspbian, and we can work from there.

Thoughts..?

[bennuttall]

In any case, I suggest the first stage is getting Mu into a complete, packaged and installable form for Raspbian, and we can work from there.

Yes :+1:

[XECDesign]

Might be worth noting early on that it might not be installed by default in our images if it drags in recommended packages as dependencies.

[ntoll]

@XECDesign out of interest... why? How's that different to any other package that requires other dependencies getting added to the distro? :-)

[XECDesign]

Other packages tend to come from Debian and have strictly defined dependencies. From the maintainer's guide, 'depends' level dependencies are much more than what QCharts is to Mu:

The package will not be installed unless the packages it depends on are installed. Use this if your program absolutely will not run (or will cause severe breakage) unless a particular package is present.

It would be absurd for, say Eclipse, to drag in every single plugin for every single mode it supports.

We want out image to fit on 8GB cards and Mu already drags over 60MB of dependencies. We left out Vim because it's too large (2MB). Small packages add up very quickly.

Okay, we may already have gpiozero and pgzero installed, but a user who may only want to print some text on the screen should be able to remove libraries they don't need should they do so.

If the package is to be upstreamed, it's better to do it right as early as possible.

[ntoll]

Hey hey @XECDesign,

Completely understand and acknowledge your points, these decisions are hard..!

60MB of dependencies...? My guess is that most of those are actually Qt (which is needed by other things you already package, like Sonic-Pi).

Leaving out Vim!?!?! That's heinous and you're obviously an Emacs person... :grinning:

Just a note on terminology... where you say "user" I imagine (in the context of Mu) "beginner programmer" -- who probably doesn't even have an opinion on what it is they want to do, nor understand how they might even remove seemingly unnecessary libraries (all worthy aims, btw, I'm just saying that the user Mu aims to help won't likely have the skills or knowledge needed to make this sort of a call). The point of Mu is that it facilitates transition: from complete beginner to someone with enough knowledge that they can graduate to a "proper" editor, IDE or coding environment.

Does this make sense? Happy to discuss further! My aim is simple, to make it as easy for beginner programmers (the target users) to get hold of Mu. I'm all ears and open to suggestions. :-)

[bennuttall]

Leaving out Vim!?!?!

Vi is included, Vim is not. Neither is emacs (think of the children!).

[ntoll]

@bennuttall phew... that's a relief to know. A unix without a vi[m] clone..?!?!?!?! Unthinkable... :-P

[ZanderBrown]

Leaving out Vim!?!?!

Vi is included, Vim is not. Neither is emacs (think of the children!).

Unpopular opinion: Nano is okay

[bennuttall]

Looking at Raspbian's Debian package for this, there are a few issues:

• guizero isn't in Raspbian, so we couldn't have that as a dependency (we'd just leave it out, but luckily this doesn't break Mu)
• We don't have the luxury of being able to match specific version requirements for dependencies - you just get whatever is in Debian - so as long as the versions of matplotlib/etc are suitable and don't break Mu or certain features, that shouldn't be a problem

Note that gpiozero and pgzero are already pre-installed, so that's moot.

So when 1.0 is ready, please let @XECDesign know and he'll update the Raspbian package accordingly.

[bennuttall]

But to get into upstream Debian I would suggest you prune the dependency list and keep it just to essential packages (which must already exist in Debian), otherwise it'll be too much work for the Debain maintainer to be able to approve, and it may never happen.

[XECDesign]

Just to clarify, I'm not in charge or what ends up going in the image, so that comment was just a side note.

As far as our (raspberrypi.org) apt repo goes, I am going to stick with actual dependencies (no gpiozero, pgzero and so on). They are already installed on the image, so there is no additional benefit to having them anywhere in the control file. If our version ends up getting replaced with an upstream Debian version which has additional dependencies, that's okay.