Pythonturtle: Build installer packages automatically

[bittner]

Following up on issues https://github.com/cool-RR/PythonTurtle/pull/105#issuecomment-421965482, #106 and #90, I'd like to start collecting information on how to (best) do packaging for all platforms PythonTurtle runs on. When we know how to perform packaging manually it will be an easy task to automate it all.

In other words, we need

1. a list of target package types (e.g. MSI for Windows, macOS AppImage, Debian DEB, RPM, Snap), and
2. a complete list of concrete, detailed steps (read: shell commands) of how each package type is built, from source code living in source control only.

NOTE: If writing this up in this issue proves uncomfortable we may want to use the project Wiki to help us move along.

[bittner]

We currently seem to use py2exe and py2app for Windows and macOS.

According to a recent talk at PyCon Otto we should switch to PyInstaller, which allows building stand-alone executables for Windows, macOS and GNU/Linux alike.

[cool-RR]

I agree we should definitely use PyInstaller instead of py2exe and py2app.

And I agree that we need a reliable and automatic build process.

[bittner]

We should also not forget about building the PythonTurtle Python package and push it to PyPI when releasing.

I'd also suggest to trigger releases by pushing a Git tag (with the version number) to the GitHub repository. The installers may then be downloadable via the release page. I hope that's all possible in a seamless way.

[cool-RR]

I agree, though that kind of automation is a bit of a high goal, if we even get to the point where we could do a release, I'd be ecstatic.

[bittner]

if we even get to the point where we could do a release, I'd be ecstatic.

Then prepare your space suit! :smile:

[bittner]

As suspected, PyInstaller will require to run on the infrastructure it should build the installer for (e.g. on Windows for a Windows installer, on a Mac for a macOS installer).

I have neither a Windows and Mac. Hence, I need to do the cumbersome finding-out how building is done on AppVeyor. Also, we need a macOS with Python 3 installed (and cross your fingers that the documentation is only outdated and we can also build with Python 3).

I'll probably best start with automating building a Python package (and upload it to PyPI automatically). IIUC, a setup.py is needed for building with (some features of) PyInstaller, anyway.
We'll have one possible way of installing PythonTurtle already this way.

[bittner]

Do you want to add me as a maintainer to the PyPI project? Would make it easier for me to automate things when I can use my own credentials. My username is "bittner", unsurprisingly.

[cool-RR]

I added you as a maintainer on PyPI.

Regarding PyInstaller, I wonder whether a VM with ReactOS would work for generating a Windows installer.

[bittner]

To make pip install PythonTurtle install the program and all needed dependencies proves to be difficult:

• The Python package wxPython seems to install successfully on Windows and (maybe) macOS, according to the wxPython Wiki.
• On Linux distributions it would be necessary to install distro-specific binary wheels -- this may be possible with setuptools-extras in a questionably usable way.

Conclusions:

• We _could_ add install_requires=['wxPython'] to setup(), but this would fail unless the prerequisites (mentioned in the README) were already installed. We may be able to catch the - very technical error - and print a helpful hint in case the installation fails.
• As an alternative, we may trust users to perform the installation steps in the README and let PythonTurtle print a friendly _reminder_ when the wx module can't be imported (ModuleNotFoundError in the application module).

Doing either in a user-friendly "auto-detect prerequisites" way using platform.release() and/or os.name would be possible but also quite cumbersome. I suggest, we simply print a text message and refer to the installation section of the README with a URL.

[bittner]

I wonder whether a VM with ReactOS would work for generating a Windows installer.

I'm not sure how we would use a VM on a service in the cloud that would build the PythonTurtle package in an automated fashion. Can you elaborate? (Do you intend for local use on a developer machine only?)

[cool-RR]

You're right, I didn't think this through.

I guess there are 2 questions:

1. Could ReactOS work for generating a Windows installer using PyInstaller?

2. Could we set it up to work automatically, on the cloud?

Let's say that the answer to the first question is yes, and there isn't a SaaS provider that gives that service. Then say we can set up a Docker image that runs VirtualBox with a ReactOS guest. Maybe we can find a SaaS provider that'll run that Docker image for us?

[bittner]

https://github.com/pypa/python-packaging-user-guide/issues/267 contains quite a few interesting links to packaging Python applications.

[bittner]

As soon as uploading the Linux bundle to GitHub is settled (via PR #123) we'll jump on the CI train for macOS, yeah! :yum:

[bittner]

The next great thing after getting EXE for Windows and bundles for macOS working could be building an AppImage for Linux. This is something similar to a .dmg file on a Mac. It would be a single solution for all GNU/Linux distros! :tada: :smiley:

AppImage also has (somewhat) an AppStore, dubbed AppImageHub. And the comparison makes you understand that it's really a cool thing. :+1:

[cool-RR]

Sounds good to me!

[bittner]

As of now, building an application bundle on macOS (using PyInstaller) is not fully working. As an alternative, I thought, we may create a Homebrew formula. I'm currently looking at the related documentation:

• Formula Cookbook -- _Introduction_
• Python for Formula Authors

Hmmm, it looks like it needs someone with a Mac to effectively develop the formula. Though it doesn't seem difficult, really. :thinking:

Here are formulae related to Python (3):

• Python (seems to install 3.7; Python@3 is a symlink, a so-called "Alias")
• wxPython (depends on Python@2, see also https://github.com/Homebrew/homebrew-core/issues/32744)

It may be sufficient to depend on Python@3 (=3.7) and install wxPython via pip (as in our tests).

[bittner]

Just for reference, if the automatic releases don't work (they probably won't currently since the api_key: is set to just a sample value), here's how the VSCodium project does automatic releases on GitHub. :+1:

[bittner]

Ram, you have mentioned "Windows installer" to be the top-priority of the present. I do agree.

Let's note down the current state of the union.

Windows Installer (progress overview)

• Windows builds are working on AppVeyor since October 2018.
• Those builds create a Windows .EXE binary and a .MSI installer (see example job).
• The .MSI installer's file size is so small that it suggests that it's actually empty.
• I recall, you have tried out the .EXE (by downloading and running it) and you confirmed that the .MSI wouldn't work. (Or was this someone else who tested it for me? Not sure.)
• I recall, I mentioned in some other issue (or was it a private chat of us?) what's needed to build a working MSI installer (it's actually just a MS Visual Studio project file, or so).

I think, I have prepared everything. Now, as I have neither a Windows computer nor I plan to buy one (same for macOS, by the way), the only thing missing is someone who takes over and puts the missing puzzle piece in place. From my gut feeling we're really, really close to "task completed".

Next steps (volunteer needed)

It looks like, all you have to do is create the necessary configuration with the Microsoft Visual Studio Installer Projects extension, according to an answer on StackOverflow.

Hence, can you take over? What needs to be done, IIUC, is:

1. Create a Visual Studio Project with an Installer setup
2. Make sure the installer builds successfully on your local machine
3. Add the .vdproj file to this project

AppVeyor will then detect that file, build the installer and provide it for download at the build Artifacts page.

References

• Software pre-installed on Windows build VMs (AppVeyor)
• Added a Visual Studio Install Project and now and getting errors during build (AppVeyor forum)
• Example of building Microsoft Visual Studio 2017 Installer Projects (GitHub repo)

[cool-RR]

From my gut feeling we're really, really close to "task completed".

I don't feel that if I'll start working on it, it'll be close to completed. I don't even know how to use Visual Studio Project, or what's our current plan. If I were to take this task, I'd guess it'd take a few days of full-time work.

You've put a lot of effort into this so I don't feel great about telling you this, but I definitely won't volunteer to work on this issue. PythonTurtle isn't close to being a priority in my life right now that I'd spend more than an hour, tops, on improving it.

Am I being annoying by pointing to this issue from other issues, saying nothing is worth anything without this issue? I won't stop you from working on any other issues, I just probably wouldn't join because that effort seems wasted to me, but I won't block you, and if you'll need me for code reviews or small things, I'll be happy to do that.

[cool-RR]

Having another thought at this. You say "I have neither a Windows computer nor I plan to buy one". If your objection to do the items you listed is because you've done enough on this ticket, or that you prefer not to spend your time doing Windows-related things, then yes, that's a fair objection.

If the issue is that you don't want to spend money on a Windows license, then maybe that's solvable. One idea would be to use ReactOS, and seeing whether that runs Visual Studio Project. If ReactOS isn't an option, then maybe I could get a Windows license/machine to you. Let me know if that's helpful.

[bittner]

Let me know if that's helpful.

Do you know a person that is _enthusiastic_ about programming on Windows? Windows is a messy environment for Python. I don't want to contribute to a false image of "Python + Windows" fit well together, that's all.

[bittner]

See https://github.com/PythonTurtle/PythonTurtle/issues/108#issuecomment-683420022 for alternative solutions to build an MSI installer (WiX, WiX.Py). We may be successful that way in combination with GitHub Actions. Does someone want to give that a shot?

[cool-RR]

Does someone want to give that a shot?

I don't, so I guess we'll wait until a possible volunteer arrives.

[cclauss]

A task list...

• [ ] __Linux__: pyinstaller build successful
  
  
  
  • [ ] Automated on GitHub Actions
  
  
• [ ] __macOS__: pyinstaller build successful
  
  
  
  • [ ] Automated on GitHub Actions
  
  
• [ ] __Windows__: pyinstaller build successful
  
  
  
  • [ ] Automated on GitHub Actions
  
  

[bittner]

A few hints to help you make progress:

• [x] With python3 setup.py clean bundle you can build all binary bundles (PyInstaller is used).
• [x] The binaries build successful for all platforms (including .exe for Windows), for what I recall.
• [ ] Windows users seem to expect a Windows installer; this could be a MSI or WiX.Py installer (both aren't yet prepared).
• [ ] GitHub Actions should automatically push binaries to our GitHub Releases whenever we push a Git tag to master; we may want to use something like actions/create-release to make this less laborious for us.

[bittner]

We received a super-nice answer by @judej on the Microsoft Developer Community request for help that I had posted! :joy:

Bottom line: No MSI volunteer for us, but there are two more interesting projects, one that builds an installer and the other that builds a Python ZIP App:

1. pynsist – _a tool to build Windows installers for your Python applications. [...] bundle Python itself, so you can distribute your application to people who don’t have Python installed_
2. shiv – _for building fully self-contained Python zipapps as outlined in PEP 441, but with all their dependencies included!_

That could be two nice alternatives worth evaluating. Thanks Jude! :100: :medal_military:

[cool-RR]

Cool. I also recently heard about PyOxidizer. I'm still not 100% sure I understood the project (discussion) but it's possible it does what we want.

[cclauss]

I will take a shot at doing this with shiv over the weekend. Please try PyOxidizer and then we can compare notes.

[cool-RR]

Sorry, I started a new job a few days ago and I won't have time to do anything :(

[bittner]

PyOxidizer sounds like an experiment to me: A Python application running on "a Rust VM", an OS-native layer created using Rust. I'm not sure what benefit this provides. Doesn't the PyInstaller setup we run do it all fine? :3rd_place_medal:

shiv is a totally new approach (new in our "portfolio" of release artifacts), which might be interesting to provide to our audience. I find the idea thrilling (even though I'd be ready to bet few people might use it). :2nd_place_medal:

pynsist sounds like the Microsoft Windows installer we're looking for. This might provide the user experience Windows folks seem to expect: :one: Download installer, :two: run it to install and add a start menu entry, :three: start application from start menu. :1st_place_medal:

[cclauss]

Downside of pynsist is Windows only. I will try to advance Shiv (from LinkedIn) on Mac and Linux.