mu 🚀 - Clean up requirements.txt / setup.py

General patterns

The following is a massive generalisation, but most Python projects usually fall into one of these two categories:

Libraries: They want to be as compatible with as many different dependency versions as possible, they are usually hosted in PyPi.
Applications: They want to have their dependencies as locked as possible. I'm not sure, but I think most of them are probably not hosted in PyPi (or very few are), instead they get deployed to a server, or somehow packaged for installation on different platforms (wheels sometimes used for this as well).

The general patterns I've noticed is that a lot of projects have unpinned versions in setup.py (or as unpinned as possible) and fully pinned versions in requirements.txt (i.e. pip freeze output) .
This is because is common to use requirements.txt for "application-like" packages, were they deploy to a server, and installing exact versions is recommended.

Our patterns

In our case I would consider Mu an application, but it is also delivered via PyPi, making it have some of the library requirements as well, which puts us in a more complicated position for this.

In our use case we need to have, at least, some minimum versions in setup.py, due to bug fixes affecting us, incompatibilities between specific versions, or new features used.
We also cannot use the latest versions of some packages because some of these dependencies are fulfilled by debian/raspbian packages.

We currently use requirements.txt as dev dependencies, and setup.py for the app dependencies. I guess we can move the dev dependencies to an extras_require and install with something like pip install -e .[dev]. Sounds like a good alternative to me, although I have not used this before, so don't know if there are any disadvantages.

Current Python packaging state

On a similar note, the state of Python packaging is always evolving, as this has been a difficult/complicated/confusing area for quite some time, and it also means there are different fronts trying to tackle some of this issues in different ways.

This is a long topic, but I would emphasise the following. I'm sure there are many more, I'm not very experienced in this area:

Pipenv

I'll start with Pipenv because I was quite exited about it and have been using it for the past few months.
General info can be found in their website, but these are my main take-aways for this project:

Made for applications, not libraries (libraries are explicitly out of scope for this project)
It also locks the Python version (and if available can use pyenv to download and create a virtualenv for that specific version), cannot select a range.
Pipfile.lock has issues with cross-platform applications with different dependencies
It replaces requirements.txt with two files (Pipfile and Pipfile.lock) but not setup.py, so we would still have 2 places with dependencies that could get out of sync.

For these reasons I don't think this is a good fit for Mu.

Poetry

I haven't used it yet, but it's definitely on my radar to try out soon. In paper sounds like it could be a good option for Mu, but it'd be great to find somebody that has used it already to see what they think.

Flit

Another one I haven't tried myself, but I've heard great things about it from different people.
Mu might not need it, as it already has a pipeline for publishing to PyPi, but could be useful if some of the changes discussed here could affect this.

Why bring this up? (pyproject.toml)

Because some of this tools have started using the new pyproject.toml file to record dependencies (not the original purpose of the file, but can be extended for these types of tools). I haven't used it yet, so I cannot really give an informed opinion, but could remove the need for a setup.py file and some of the issues that it has.
From what I read I like the idea, but to be fair I also really liked pipenv in theory and it wasn't until I started using that I could see the warts.

carlosperate on 19 Feb 2019

@carlosperate's comment above sets an excellent backdrop against which any changes should be considered. I find it very valuable and it's a topic I periodically revisit when it comes to setting up, maintaining, and distributing Python based projects.

Thoughts

The "Libraries" vs "Applications" along with "loosely defined dependencies in setup.py" for the former vs. "locked dependencies in requirements.txt" for the latter are ideas I generally agree with.
More importantly, however, is the fact that setup.py is a tool used for the distribution of a given project and, in particular, but not exclusively, to support distribution via PyPI and later installation via pip.
Whether or not (or how much) to lock Mu's dependencies in setup.py is up to how certain we want to be about the result of a pip install mu-editor: what if it brings a newer PyQt version that somehow breaks things? On the other hand, would we want to lock the dependency to a specific PyQt version, preventing users from benefiting from an updated, better, faster, cheaper, more secure PyQt version?
The current ways Mu is distributed are far from simple:
- Packaged Windows installers.
- Packaged macOS installer.
- Both of the above include:
- A Python interpreter.
- A bunch of nice-to-have (?) third party libraries like requests, numpy, matplotlib, etc.
- PyPI pure Python wheel and sdist.
- Debian/Raspbian DEB packages with dependencies set per the platform's APT/DEB tooling.
- Others I'm not aware of? :)
To make things a little less trivial, we then have a few variations WRT dependencies:
- On Raspbian (arm only or x86 too?) there is no plotting support library, IIUC.
- For PyPI distributed Mu some dependencies depend on the Python version (example: no black if Python version <3.6).
Regarding tooling, I think we all share a dose of "pain" when it comes to packaging and distributing Python projects to an open-ended audience: especially with applications. Having said that, I'd tend to go with what the team around Mu feels comfortable with, mostly looking for a good balance of effort and sustainability: if setuptools works maybe there's no point in investing time learning and adapting to some other tool that, who knows, may still be in its infancy and changing a lot in the near future (nota bene: I'm not arguing that any tool you mentioned is, ok? I just know they are "new kids on the block", taking different approaches at the problem, which at first sight might look appealing but later be found limiting in some ways; the fact is, like you mention, that the landscape is changing and the PyPA has been pushing things forward a lot, lately).

All of this in other words: the more time we can have contributors focusing on Mu itself instead of on tooling around Mu the better. Sustainably. :)

Objectives

I believe we should start out by enumerating explicit objectives we may be looking for as a result of this process. From there we can discuss whether or not they are attainable and at what cost. Here are a few I find generally valuable:

Have a single source of truth regarding dependencies.

A quick grep points to four such sources, as of now: requirements.txt, setup.py, win_installer32.cfg, and win_installer64.cfg.
Such source of truth should handle the platform and Python variations I mentioned above.
For developers, given an isolated Python environment and a source tree:
- pip install . should install Mu, grabbing just its runtime dependencies from PyPI.
  With this, given that there's a console_script entry_point in setup.py, typing mu-editor in the CLI should launch Mu.
- pip install -e .[dev] should install Mu in editable mode, grabbing not only its runtime dependencies from PyPI, but also all of those related to development: test runners, doc builders, Windows/macOS packagers, you name it (everything a developer could ever need to develop).
There could be lighter variations on the previous example with, say, pip install -e .[test] installing only the runtime + testing dependencies, or pip install -e .[doc] installing only the runtime + doc building dependencies, etc. (example here where dev includes tests and docs)
- Either of these should properly cope with platform / Python version dependencies.
- Building wheels and sdists should be a single command away.
  Today we're at python setup.py sdist bdist_wheel which meets the objective.
- Building packaged Windows / macOS versions should, again, be a single command away, where regardless of the tooling/procedure, the dependencies would be sourced from the single source of truth I initially established as an objective.
For the benefit of end users:

(preamble: I'll assume that end users will mostly get Mu from the Windows / macOS packaged installers and from Debian/Raspbian APT repositories and that those going for pip install mu-editor are the ones I'd consider i) no longer the target audience for Mu, maybe needing explicit guidance in such process, or ii) advanced enough that they're "at home" with the command line, pip, and friends).
- Everything still works just as it does today, only better. :)
- Maybe, now that #749 has been merged, the packaged Windows and macOS installers could avoid including/shipping third party libraries, trimming down the distribution file / installed size. This must be thought over because users updating from 1.0.2 could see their programs failing to import, say, requests, on a newer, trimmed down Mu version distribution.
- Advanced users would have a solid and consistent experience with pip install mu-editor, regardless of the platform and Python version (not sure if that is a fact today, I suppose it is).
- Setting up a development environment is simpler and less brittle so:
- Advanced beginners will have an easier time exploring the idea of changing and contributing to Mu.
- All users indirectly benefit because developer's lives are easier.

Call to Action

The objectives I shared mostly boil down to / derive from "having a single source of truth" regarding dependencies. Most of the others could be considered the result of my own experience and perspective on the topic which I'm more than happy to discuss.

Which other objectives would others consider?
Which of the ones I mentioned make no sense?

PS: These changes can become somewhat messy, affecting development, CI, packaging, distribution, everything. So I'll proceed with care. :)

tmontes on 19 Feb 2019

Hi Folks,

Some context from me.

Packaging is hard and part of the biggest delay for the release of Mu 1.0 was simply trying to build a pipeline for creating installers for Windows and OSX. This proved to be extraordinarily hard and I don't want to throw away this work.

As for the single source of truth... yup, that's a good idea, except there are packages needed for Windows (pynsist) or Mac (briefcase) in order to package Mu properly.

I deliberately pinned versions in setup.py because we encountered a case where a new version of PyQt5 broke with another module (since fixed). By pinning versions we are acting conservatively and guaranteeing folks get a version that will work. However, with 1.1 coming along these versions should, of course, be updated to the latest and greatest (including updating the version of Python to 3.7 where this is included in an installer).

School network administrators never use pip to install anything. They want a Windows installer that comes with everything which they can use to mass-install an computers in their labs. Ensuring that the installers work for Windows and Mac (the two most popular platforms which beginners use with Mu) is THE MOST ESSENTIAL task when it comes to packaging.

I like the distinction introduced by @carlosperate between application / library. When I've worked on large web applications we always made sure we pinned our versions so we could be certain deployment to new servers would recreate exactly the correct environment.

I'm not keen on using anything other than the LEAST SURPRISING tools for packaging Mu for pip installation. I've been involved in Python for 12 years now, and packaging has always been and continues to be a complete mess and pain in the neck. In this situation I retreat to a position of "least surprise" and "KISS". Our Makefile contains all the commands we need for building Mu.

This leaves how we deal with requirements and versions... I agree with @tmontes that having a single source of truth (with perhaps "add-ons" for platform specifics) is a good idea. Now that support for third party modules has landed in master, we can also throw away those "nice to haves" we packaged for folks.

As a developer my expectations have been that requirements.txt contains the stuff I need to develop the module, whereas setup.py is how I might install the module without needing to develop it. This feels like the other way round to how you folks are discussing it. I honestly don't mind which... ;-) so long as it's easy, unsurprising and simple. :-D

Ugh... packaging.

ntoll on 19 Feb 2019

👍2

Yeah, I think it's a good call to use what's still the most common and simplest approach.
I would love a future where pyproject.toml is the single source of truth for a bunch a different packaging and deployment tools, but we are not there yet (or who knows if we will ever be).

As a developer my expectations have been that requirements.txt contains the stuff I need to develop the module, whereas setup.py is how I might install the module without needing to develop it.

Yep, this is my general view as well. I think by having an extra_requires in the setup.py file instead of a requirements.txt file, the only change would be to switch from pip install -r requirements.txt to pip install -e .[dev].

It's not as obvious, and perhaps not the first thing seasoned developers type after a git clone, but I think it's seen around enough that it doesn't feel too alien.

As far as platform differences go, they can be implemented using with environmental markers.

I think we can also relax some of the current pinned versions. pycodestyle pyflakes and pyserial could probably be fine with >= operators instead of ==.
Might be a good idea to latch all of them to the same semver major version (so foo>=2.3,<3.0).

Now that support for third party modules has landed in master, we can also throw away those "nice to haves" we packaged for folks.

I think it's a good idea to remove any non-essential packages from Mu for the alpha/beta testing and request to teachers if they can try to install new packages.
To be honest I'm a bit worried proxies or intranet settings will block this feature, in which case we might have to put those back into the installer. So, if we remove them during testing it's more likely we can get the feedback we need to fix that situation.

carlosperate on 19 Feb 2019

@carlosperate +1

Maybe I'm showing my (old) age... but I wasn't aware of pip install -e .[dev], I've always just done pip install -r requirements.txt. ;-)

ntoll on 19 Feb 2019

[dev] is basically just any name you want to give to an entry in the setuptools extra_requires.
It's a pattern I've seen before, but it is not used that much, just like some people use requirements_dev.txt.

https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies

carlosperate on 19 Feb 2019

Maybe I'm showing my (old) age... but I wasn't aware of pip install -e .[dev], I've always just done pip install -r requirements.txt. ;-)

They're different things at a fundamental level: the former installs "the project" (+dependencies), the latter installs "dependencies".

Correctness and completeness note

For the sake of correctness and completeness, I suspect a requirements.txt file could be created that would also install "the project", but I'm not sure I've ever seen one.

Let me try to break it down a little more, in simplified terms, trying to avoid the complex (complicated?) nature around all of this and the temptation for too many side notes. :)

`pip install -r requirements.txt`

Collects packages per the specs in the file, resolves dependencies, installs them such that they become importable.

The project itself is not installed, meaning, among other things, that:

The package is not importable because it was not installed in a directory in the Python import path unless, of course, the current working directory happens to be the root of the project which contains the mu source package directory: itself a source of potential pain that comment on lightly below.

Console scripts, like the mu-editor command specified in the project's setup.py are not installed and, thus, not runnable.

Note on dependencies

Universal dependency resolution is a non-trivial computational problem and, AFAICT, pip doesn't handle all possible scenarios.

Note on project layout

Why it may be a good idea to have the source code of your project under a generic src directory has to do, among other things, with tests. Such layout ensures that the project is not importable at the root unless it is installed. This means that tests must run against an installed version of the project (a good thing) instead of against the non-installed source version (which may pass, but fail to reveal installation issues, that end up failing on deploys).

`pip install -e .[dev]`

This is a variation on a simpler pip install path-to-project where path-to-project is the path to a directory containing a setup.py file, which pip will execute to install the project: I guess along the lines of python setup.py build followed by python setup.py install.

The . in pip install -e .[dev] is the path to the project, assumed to be the current working directory, pretty much as in the case of pip install -r requirements.txt.

When installing from source -- either from a project path as in this example or, say, from a so called sdist .tar.gz file -- pip does not determine the project dependencies; instead, it executes setup.py which, most probably built on top of setuptools, determines its dependencies from the install_requires argument of setuptools.setup. With that, setuptools collects packages, resolves dependencies, and installs them before proceeding to install the "project".

The "Note on dependency resolution" applies in this case too, AFAIU.

The [dev] after the path-to-project is an optional component of a package specification which, if I'm not mistaken, is defined in PEP 508. It supports requesting "optional extra features" from a given package. Among other things, from a packaging standpoint, such "extra features" are freely defined in a project's setup.py via the extras_require argument of setuptools.setup that establishes additional dependencies for a given "extra". At install time, the full dependencies are determined from the the union of the installation dependencies set by install_requires with the extra dependencies set by extras_require.

Lastly, the -e in pip install -e .[dev] installs the project in editable mode. In other words, it does not copy the needed Python modules/packages to the Python environment's site-packages while having it installed as if it were: this supports the typical edit/test development cycles without the need to transform it into a less productive edit/uninstall/install/text cycle. But I suppose you were aware of it already, given that it is mentioned in the docs here (even though I don't fully understand why).

Note on wheels vs sdists

One of the benefits of wheels is that installing them does not require executing the project's setup.py on the target systems. Who knows if a given project's setup.py includes malicious code? Do you audit every project you install from PyPI to see if it's a wheel or sdist and, in the latter case, inspect setup.py to check for malicious things? Not to panic, but to be aware of, and be taken seriously.

Wrap up

I've written these words sharing my understanding of the topic, such that we call all be on the same page, as much as possible. They're shared humbly, hoping they may be of value (NB: I wouldn't call myself an authority on packaging: not by a long measure!).

I've been thinking and reading a bit about this specific Mu challenge during my breaks and I think I'm almost ready to write a few ideas on "how to move forward": simplifying things while still being flexible and, importantly, minimising risks associated to change. :)

I'll be back soon.

tmontes on 21 Feb 2019

There's one thing I'd like to understand, before wrapping my ideas around the whole thing and trying to come up with a possible solution.

AFAICT, the packaged distributions are created:

With briefcase for macOS.
With pynsist for Windows, in their 32 and 64 bit variants.

One side-consideration

Each of these dependencies is currently specified in their own way. This could also be improved, for consistency and more benefits:

The pynsist dependency is defined in requirements.txt, including a useful environmental marker. It currently reads pynsist; sys_platform == "win32".
The briefcase dependency is hidden away in .travis.yml and it took me a while to find it. I ended up grepping Travis logs until I found it in a single, isolated pip install briefcase command.

Note how a developer can trigger a packaged Windows build locally with (relative) ease, but the same cannot be said for macOS. Not very serious, no big impact: just another thing that can be improved (and the easier for contributors to build their own packaged distributions, the more the whole process is exercised and tested, which is a good side-effect of this improvement).

The key question

AFAICT, briefcase can package Windows applications too. Apart from the different native dependencies [1], wouldn't life be simpler if a common tool was used for packaging across macOS and Windows?

Given that briefcase isn't currently being used for Windows packaging, I assume that this has been considered and that some limitation was found. Can you, @ntoll (on anyone else), share any details on this please?

Thanks!

[1] At a glance, briefcase depends on WIX while pynsist depends on NSIS.

tmontes on 22 Feb 2019

@tmontes so the whole packaging story with Mu is full of pain, frustration and many abandoned solutions.

The order of packaging solutions was Pynsist for Windows, then several months of trying to find a solution for OSX and eventually (after taking part in the BeeWare sprints at PyCon 2018) discovery that briefcase had recently been updated to meet our requirements. This explains the two solutions.

I agree that having a single solution would be preferable, but in this case (and one day I'll show you the scars on my back from the packaging pains) I feel it's definitely a case of "if it ain't broke, don't fix it". By all means go ahead and see if it's possible to settle on a unified briefcase solution, but "here be dragons" is all I'd say on that matter. ;-)

ntoll on 22 Feb 2019

Also, it should be relatively trivial to add a make mac command to the Makefile to kick off briefcase.

ntoll on 22 Feb 2019

@ntoll, like you, I do understand each and every one of your words. Down to every drop of cold sweat I know wants to set itself free from our bodies when we even whisper "automating native Python packaging"... I have also travelled such dangerous and Dragon filled roads myself, a few times before. :)

Thanks for the quick feedback. I now have all the information I (believe) I needed. I'll let it cool down a bit in the back of my mind, and write up a possible solution (and yes, I'm already seeng that silly Pink Dragon around that next corner! I wonder how many more and how horrible they will be...).

I will thread carefully and, as always, will welcome discussion.
Drama aside, I do believe we can do simpler and better within a nearly zero-risk scenario. But I'll have to do it, see it, run it, before making any definitive claims.

PS: I even have 4 dev environments I can test solutions on: macOS, Win 10, Raspbian, and Linux! :)

Also, it should be relatively trivial to add a make mac command to the Makefile to kick off briefcase

Sure! But doing that now would precisely highlight the dependency declaration inconsistency we have been discussing: would make run pip install briefcase by itself as is the case for the current Travis config? Or would such dependency be prepared along with the "full/proper/correct build environment"? :)

tmontes on 22 Feb 2019

Yup... I can't argue with any of the above. This is a gnarly problem... and if you want to do a "St.George" (patron saint of both England and Portugal I believe) then by all means don that armour and go hunt dragons. :-D

The "messy" current state is basically down to the historical reasons I outlined and, TBH, if we can simplify/unify things, that would be a good thing :tm:. I will, of course, be cheering on any efforts in this direction!

ntoll on 22 Feb 2019

@ntoll, Dragon ahead warning:

The current master fails to be properly packaged on macOS with briefcase.
Motive: this commit added an environmental marker to install_requires in setup.py which, AFAIU, briefcase is not handling (but pip, for example, handles it as expected).
This seems to result from a setuptools way of doing things, IIUC.
Possible ways forward:
- Improving briefcase to become environment marker aware (not sure if setuptools provides a good enough code base to handle that easily).
- Avoiding environment markers :( and handling them procedurally in setup.py, dynamically adding dependencies to install_requires based on conditionals.
Won't bother you with further details (ask if interested).
Clearly needs more investigation!

PS: I found this out while exploring a few ideas, nothing complete to share yet.

tmontes on 24 Feb 2019

Aha... so we should let the upstream BeeWare project (who maintain Briefcase) know about this.

ntoll on 24 Feb 2019

Explored a bit more. Even though I haven't fully grasped the whole of setuptools thing (does anyone?) :) I guess I figured out a way of teaching briefcase how to handle environment markers. :)

I'll raise the issue there, including a PR proposal... Let's what someone with a deeper setuptools understanding thinks of my take, as a possible solution.

tmontes on 24 Feb 2019

Like you might have been aware, I've been exploring this issue. :)

(my progress and experiments can be found in my fork's branch here)

Here's a progress report, raising some topics looking for feedback, below:

I have created a setup.py that:
- Does not import mu.
- Includes declarations of required packages, including environment markers.
With that, pip install . or pip install -e . on the project's root properly installs a working Mu copy (lightly hand tested) on macOS, Windows, and Raspbian.
Also, with briefcase master branch, a working macOS packaged Mu binary is successfully generated and works (lightly hand tested, but Tk works, REPL works, debugger works).
For completeness, I explored using briefcase to generate Windows packaged binaries: it did not work "out-of-the-box". Shouldn't be difficult to make it work properly, but I will not invest more time in it, given that we don't want to be changing everything at the same time. :)

There is more work to be done with setup.py regarding non-runtime dependencies such as those supporting test running, linting, docs building, etc., but that's the easy part which I'll address last.

Dragons ahead:

I suspect macOS briefcase packaging with Python 3.7 will need work as a properly versioned Python distribution is not readily available for it to use (didn't try yet, have been using 3.6.8).
Version locking needs thinking and decisions:
- setup.py can go either way: free, locked or, like @carlosperate suggested, latched to a given semver version.
- macOS packaging with briefcase will go along with that, just fine: at packaging time briefcase collects the matching dependencies from PyPI.
- Windows packaging with pynsist, however, needs locked dependencies it its configuration file (currently the win_installer*.cfg files in the repository root). I am foreseeing an improvement in the automation such that these files are generated at packaging time and deleted afterwards: the single source of truth being setup.py.
- Wheel generation will include the free/locked/latched dependency information as specified in setup.py, so no problem.

The fundamental issue then becomes: do we want to ensure strict dependency consistency across the various artifacts (Windows packaged, macOS packaged, PyPI wheel, PyPI sdist) or not? If setup.py declares non-locked/pinned dependencies then there is no way to achieve that because the Windows/macOS packages will collect their dependencies at packaging time and there's no guarantee that between one Windows AppVeyor packaging job and another Travis macOS packaging job the same exact PyPI dependencies will be collected.

AFAICT, there are no mechanisms in place today that ensure such strict dependency consistency. I'm not sure what to think of this... While locking dependencies helps ensuring things work and facilitates diagnostics, it may become unnecessarily limiting on one hand, and need setup.py fiddling around the release process like: lock dependencies before release and tag a version > commit > package and release > unlock dependencies after release > commit.

I don't know... What do you gentlemen have to say about this? :)

tmontes on 25 Feb 2019

I don't have much time today, but just a quick things that might help:

I suspect macOS briefcase packaging with Python 3.7 will need work as a properly versioned Python distribution is not readily available for it to use (didn't try yet, have been using 3.6.8)

The current Python 3.6 portable executable comes from: https://github.com/mu-editor/mu_portable_python_macos
I can't remember exactly what was the issue but there was a problem with the default image and this one (which we were looking into in parallel with briefcase) worked well, so the final version ended up using briefcase pointing to this zip file instead of the default ones.
The build scripts can be easily updated to create 3.7 images (if there are no issues), but that is usually easier said than done.

Windows packaging with pynsist, however, needs locked dependencies it its configuration file (currently the win_installer*.cfg files in the repository root).

I didn't realised this was a requirement from pynsist, so you can only have fully pinned (==) versions in the config file? It doesn't take < or > operators? Do we know why that is?

carlosperate on 25 Feb 2019

@carlosperate, thanks for your feedback.

...
The build scripts can be easily updated to create 3.7 images (if there are no issues), but that is usually easier said than done.

...I had this much figured out, that's why I included that note: 3.6 packaging is working now, with no efforts, other than using the latest briefcase from master. 3.7 will need the preparation of a Python distribution much like you point out. :)

I didn't realised this was a requirement from pynsist, so you can only have fully pinned (==) versions in the config file? It doesn't take < or > operators? Do we know why that is?

Essentially because pynsist is very naive WRT to fetching dependencies from PyPI. Unlike briefcase which is built on top of setuptools, pynsist has its own implementation where, at some point, we can find this:

https://github.com/takluyver/pynsist/blob/master/nsist/wheels.py#L73

I haven't explored much further though, so I may be a bit off. :)

However, I was anticipating that and, like I mentioned earlier today, I have a working solution for that challenge nearly ready, regardless of having setup.py's dependencies locked, or not! :) Stay tuned.

tmontes on 25 Feb 2019

👍1

Gentlemen, I bring you good news, after having Slayed a few funny dragons along the way. :)

In short, my work-in-progress branch now:

Has all dependencies declared in setup.py, along with the respective environment markers.
Contains no other sources of dependencies: requirements.txt is gone, much like the win_installer*.cfg files.
Installs and runs Mu successfully after a pip install -e . into a fresh virtual environment: on macOS, Windows, Raspbian, and Linux.
Successfully generates working 32 and 64 bit Windows installer packages with the make win32 and make win64 commands, as before.
Successfully generates a working native macOS application with a new make macos command (this one still depends on an future briefcase release which I asked @freakboy3742 here to hold until we're certain there are no other improvements needed).
The documentation is updated to reflect the changes.

What's pending before I issue a PR for evaluation (of course, the branch is there for interested eyes):

Updating the Travis and AppVeyor configuration files (which is actually where all the fun starts!).
Addressing a pycodestyle/pyflakes complaint WRT the simple and readable approach I took at not importing mu in setup.py (argh!). :)
Clarifying a few details, maybe with your help:
- Why is the simplegeneric package needed? It is explicitly installed in .appveyor.yml with pip install simplegeneric. It is also declared in the packages section of the win_install*.cfg files. In my branch, I'm generating working Windows installers without it. I'm probably missing something... Any help?
- Why is the dmgbuild package installed explicitly in .travis.yml? It's done right after explicitly installing briefcase but it doesn't seem to be used... Help? :) PS: I see that it is referenced in the Release Process docs, but fail to understand why it's installed at Travis. Second PS: I just hit this dmgbuild issue!... :(

Your feedback will be very much appreciated. :)

tmontes on 27 Feb 2019

👍1

Well I'm very interested (and v. grateful that you've taken the time to try this out). However it's not come at a good moment timewise for me. But I do hope to find the moment to try it out as soon as I can.

tjguk on 27 Feb 2019

@tjguk, thanks for your note. Please to not feel obligated to try it out in a rush... The fundamental purpose in my updates is sharing progress and questions, as I move forward. I'm not in a hurry here, I'm just "in the zone". :)

tmontes on 27 Feb 2019

I just did it: PR #779 is going through its tests (AppVeyor will pass, because it already did in my fork/branch). Please refer to it for details on what is changed, how and why.

Happy to clarify, discuss, review, etc. Thanks!

tmontes on 28 Feb 2019

All,

PR #779 has just been merged. There may be valuable follow actions / tasks that I enumerated in the post merge comment, there.

In short:

requirements.txt and friends are gone.
Use pip install -e .[dev] instead.

...wondering if we should add a requirements.txt file like:

# QUICK INSTALLATION
# ------------------
#
# To install, run "pip install ." including the trailing dot.
# This installs Mu and all its runtime dependencies.
#
# To setup a development environment, run "pip install -e .[dev]", instead.
# This installs Mu and its runtime dependencies, as above, along with all the
# development ones.
#
# In either case, typing "mu-editor" should launch Mu.
#
# For more details, including platform specific notes on the Raspberry Pi,
# please refer to https://mu.readthedocs.io/en/latest/setup.html.

:)

Other than that, and once developers get a handle on this renewed approach at setting up their development environments, I say this issue can be closed.

Thoughts?

tmontes on 26 May 2019

Pinging others:

Should we add the "comment only" requirements.txt like I noted above as a friendly note to those minimally experienced in Python but not familiar with installing a setuptools-based local distribution?
Or should we not, and close this issue once and for all?

tmontes on 16 Jul 2019

+1 for the "comment only" requirements.txt.

One observation about the suggested content, it does not have the quotes around the .[dev] piece. i.e:
pip install -e ".[dev]"

ukBaz on 16 Jul 2019

I'm basically in favour. My only concern is that anyone blindly doing "pip install -r requirements.txt" might miss the fact that nothing actually happens. Frankly it's a tiny risk and the next thing they'll do is to open the file to see what's there...

So I say: go ahead, obviously address @ukBaz comment about the ".[dev]" thing

tjguk on 16 Jul 2019

👍1

@ukBaz, @tjguk, just created PR #898, adding the comments only requirements.txt: your feedback there is very welcome.

Thanks.

tmontes on 17 Jul 2019

Mu: Clean up requirements.txt / setup.py

Most helpful comment

All 27 comments

General patterns

Our patterns

Current Python packaging state

Pipenv

Poetry

Flit

Why bring this up? (pyproject.toml)

Thoughts

Objectives

Call to Action

`pip install -r requirements.txt`

`pip install -e .[dev]`

Wrap up

One side-consideration

The key question

Related issues