Julia: MSIX installer and Windows Store

[davidanthoff]

I've created a MSIX installer for Julia, with the eventual goal to put Julia into the Windows Store. The whole thing is very experimental at this stage, has major gaps, needs a lot of feedback from the whole core team and if we think this is overall a good idea will take quite a bit more work to bring over the finishing line.

This will be a slightly longer post, so bear with me :)

Background

MSIX packages are the type of installer that is used by the Windows Store. In general, it seems to be where MS is going with setup technology.

At a very high level, a MSIX package is a file format for packaging up applications. There are two ways in which you can install MSIX packages: a) you submit them to the Windows store and then it gets distributed via that, b) you use AppInstaller (part of Windows) to install a MSIX package from a URL, file share etc. In both scenarios you get auto-updating of your application.

MSIX also has a notion of optional packages. Think main package for an app, and then optional packages to install additional features on-demand for an application. This is used say in games where you have an in-app purchase option to add features, those typically come as optional MSIX packages. I'll describe a little later how I'm using optional packages in my prototype.

Benefits

Here are some major benefits one gets from packaging an app in an MSIX installer:

Auto updating

This one is easy enough: one gets a fully automated updating experience for the app :)

Windows Store

Also easy enough: MSIX installers are the only way to get an app into the Windows Store. I think having Julia in the Windows Store would be a major, major simplification for beginner users.

Putting Julia.exe on the PATH

MSIX installers provide a simple and elegant way to add commands to the PATH. The way this works is that the PATH environment variable is actually _not_ modified at all. Instead, by default, the folder ~\AppData\Local\Microsoft\WindowsApps on Windows is listed in the PATH env variable, and then AppInstaller or the Windows Store can automatically add a symbolic link to our Julia.exe in that folder. All we have to do is declare in our MSIX installer that we want julia.exe on the PATH, and then the rest works by itself.

This is better than modifying the PATH itself for a number of reasons: 1) it all works without any restarts at all: if you have a shell open, then install Julia, you don't even have to restart the shell to pick up some new PATH configuration, once Julia is installed, it just works. 2) there is no risk that things get left on the system with respect to the PATH if the package is uninstalled, 3) no worries about PATH becoming too long, 4) we can julia.exe on the PATH, without putting the folder with various DLLs on the PATH.

This would also mean that novice users don't have to do any manual configuration to get julia onto the PATH, which at least in my experience has been a major hassle in a teaching environment.

Multiple Julia versions

One concern with an auto-updating installer is that one loses control over what Julia version one is using. Forced auto-updating to the latest Julia version seems a bad idea. So this is where my little project went a bit out of control... I ended up with the following design:

The main Julia MSIX installer package delivers two binaries: 1) a julia.exe that gets put onto the PATH, but really is just a launcher for the "real" julia.exe, and 2) a juliaup.exe (in the spirit of rustup) that one can use to manage which Julia versions are installed on a system.

So the main Julia MSIX installer package actually does _not_ contain any real Julia version. Instead, each Julia version is an optional MSIX package. So we have a Julia-1.4.0, a Julia 1.4.1 etc. optional package. The juliaup.exe is used to install and uninstall these optional packages. It supports commands like juliaup add 1.4.2 (installs Julia 1.4.2), juliaup rm 1.4.2 (removes Julia 1.4.2) etc. Another command is juliaup setdefault 1.4.2. This command configures which of the many installed Julia versions should currently be used.

Starting julia.exe now simply looks at the default configuration for the Julia version, checks whether that is installed, and then launches that specific Julia version.

The final piece in this is that the Julia MSIX installer (that only includes the launcher and juliaup.exe) also, by default, installs one of the optional Julia packages, currently Julia 1.4.2. So if a user installs the main package, then Julia 1.4.2 is automatically installed and configured as the default. So at first install one does not have to start juliaup at all to install an initial version or configure anything.

Major problems

There are a bunch! Some of them I think have the potential to be deal breakers, unless we find some way around them. I have ideas for most of them, but here we go:

• IJulia doesn't work. There is a lot more information in that linked issue. Short version is that I think we could get around this if we put the installer into the Windows Store _and_ MS gives us some capabilities that installers normally don't have. The infrastructure for that is all in place, this really depends on a policy decision by MS. I'm fairly optimistic that this might work, I've been working with an engineer at MS and he has been really incredibly helpful, so the general vibe I'm getting from MS is very supportive. I don't think there is any way to solve this for the AppInstaller case, though.
• Updates are broken. There is a bug in the currently shipping version of the AppInstaller that makes the auto-update feature only work after a system restart... MS is aware of that and I assume they will fix this with an upcoming patch, so one option is to just wait. This would also be solved if we put the whole thing in the Windows Store, because then the AppInstaller part (where the bug is) doesn't come into play.
• Some random other issues that we can probably work around, see here for my current list.

Major design decisions

I think the biggest design decision is whether we are generally ok with the whole juliaup design and whether we are ok that juliaup uses a Windows only technology under the hood.

I think if we were to go with this solution, the eventual goal would probably have to be that on other platforms we try to create a similar experience, i.e. that juliaup is also available on at least Mac and probably all platforms. Under the hood it might use a different mechanism to actually deliver Julia versions (i.e. not MSIX but either a platform specific tech for Mac and Linux, or something home grown).

I _do_ think there are major benefits to using the optional package feature from the MSIX stack on Windows. For one, it makes everything incredibly simple. There is hardly any code at all to get the whole juliaup story going, most of the complexity of such a solution is handled by the MSIX engine. We also get a really nice integration with the Windows UI. For example, the app part in the settings will show the optional packages as optional packages for the main Julia package, and users can uninstall them from there directly. There is also a ton of deployment tools on Windows that all work with MSIX packages, so I think this might be the easiest way to e.g. deploy Julia on say on university managed Windows machines for students etc.

But, I feel this part really needs a lot of thinking.

Another thing to keep in mind is that this is Windows 10 only, so we would need to keep the existing installer around for older platforms. I'm not too worried about that, but others might feel differently.

Try it out

You can give the current version of all of this a test run. It is not in the store, but uses the AppInstaller way of distributing things. Here is how you do it:

1. Make sure no julia.exe is on your PATH.
2. Head over to https://winjulia.s3-us-west-1.amazonaws.com/index.html.
3. Click install and follow the instructions.
4. Once everything is installed, you should be able to run julia and juliaup from any shell. juliaup should output instructions on how to use it.

The code for the whole MSIX installer is at https://github.com/davidanthoff/winjulia.

Next steps

I think this will all require time to sort out. So right now I'm mainly keen to hear what others and the core think. Is this a good direction? Is there interest in pursuing this? Oh, and of course it would be good to hear whether things work for folks :)

If we feel that is worth following up with, we can figure out the next steps. I think it would probably entail me continuing to work with the engineer at MS to bring this whole thing into the Windows Store.

[ViralBShah]

Is this different from winget?

[musm]

Is this different from winget?

Completely different.
winget is just a MSFT sanctioned chocolatey 'clone' with a better design.

[ViralBShah]

Ok, but is it then useful to have both?

[musm]

The two are not mutually exclusive. julia is already on winget https://github.com/microsoft/winget-pkgs/pull/2418

[davidanthoff]

You'll be able to install store apps with winget, I believe that is one of the next features they are adding right now.

If we do go with a Windows Store story/MSIX, it would make sense to have winget install julia via the store app, not via the inno setup. Would be trivial to switch that over.

[ViralBShah]

Ok, makes sense. If we get this all going, then perhaps we can drop the inno setup and only distribute the portable binaries on the website.

[musm]

Ok, makes sense. If we get this all going, then perhaps we can drop the inno setup and only distribute the portable binaries on the website.

Eventually yes that would make the most sense. In the meantime, we would promote the MSIX installer and portable .zip package for special cases and leave the Inno for legacy systems.

[rapus95]

I think if we were to go with this solution, the eventual goal would probably have to be that on other platforms we try to create a similar experience, i.e. that juliaup is also available on at least Mac and probably all platforms. Under the hood it might use a different mechanism to actually deliver Julia versions (i.e. not MSIX but either a platform specific tech for Mac and Linux, or something home grown).

As I'm obsessed with trying to solve every Julia related problem in Julia itself, I thought about creating a julia based updater multiple times. The obvious advantage would be the independency of the platform. For the current state of the idea I would also need a non-julian file (a proxying julia.exe) that works the same as yours, i.e. that it reads from a config which version to start (presumably support some command line arguments like --version=1.4.1) and passes all other arguments straight through to the actual julia.exe. For updating I imagine a Julia script which simply downloads the new version (maybe even via a new pkg server sub path) and simply updates the config file the proxy julia.exe relies on. In general, I mean, what's the problem with having the whole update functionality accessible from within Julia? (maybe in Pkg.jl)

[musm]

With the MSIX installer, you need to call Windows API which are easiest to access through c++and c#. It might be possible to have some of the juliaup core code in a julia package and call that through c++. I'm not sure how hard or easy it is to do that. One tricky thing would be that the juliaup core would have to be in-place update as it relies on a Julia runtime. It might also make the whole juliaup process slower due to precompilation unless it's all precompiled.

[rapus95]

@musm as we intend to support having multiple versions installed at the same time I'm not sure if we even need to upgrade inplace. Instead we can just install the new version, trigger the new version and from there delete the old one. Alternatively though, there's an easy way in windows to upgrade inplace: Just move the original files away into some OS-managed temp directory for example (that's possible even while the file is locked) and put in the new files.
As the general question, is there anything against shipping new versions over the julia package server? (And at some point maybe even have it integrated with BinaryBuilder) (as I said, very julia centric xD)
So, what if:
the Julia version would be managed by the Pkg Stdlib? Is it possible to abstract the Julia syntax, compiler & fundamentals away into some fundamental package which can be updated over the package manager?

[davidanthoff]

How do we move forward with this?

I think the next step would be to try to get this version into the Windows Store and start testing how things work with that environment. That will require sorting out who should appear as the publisher of the app in the store, my best guess right now is NumFOCUS maybe? It needs to be a legal entity that can get a code signing certificate. There are also next steps on the code for e.g. juliaup, but I think it would make more sense to first try whether the current prototype works in the store before we invest more time in fine tuning things.

I'd be happy to push this forward and see the whole store submission through, but only after we have come to some broad agreement whether the approach here is something that is acceptable and the core team is on board with it. Or to say it differently, I wouldn't want to spend a lot of time on the store submission process (which I expect to be somewhat painful and involved because we need various exceptions) only to have a discussion after that whether the design of juliaup here is ok or not. I think we should be able to come to a decision on that based on what exists today and before the store submission.

This is obviously not a niche question and probably would need a nod from many of the core folks? Who would that be? @StefanKarpinski could you help me manage that process, as the official "I get binary files onto user machines"-guru?

[StefanKarpinski]

@StefanKarpinski could you help me manage that process, as the official "I get binary files onto user machines"-guru?

I think you have me confused with someone else 😬. @staticfloat maybe?

I don't know squat about Windows and have gotten pretty frustrated over the years with the seemingly endless discussions about how to improve Windows things that never seem to actually go anywhere. The first and only Julia issue that I've muted is the one about bundling a better terminal. If Windows is going to be better, Windows users need to step up and improve things. @musm has been doing great work, but he needs some help.

[staticfloat]

I don't have strong feelings on this; my gut instinct is to just empower people who are dealing with the rough edges of windows installation (e.g. @davidanthoff and @musm) to do what they think is best. In fact, I would actually push against the assertion that we need a consistent juliaup experience across OS's and would say just build something fantastic on Windows, and if it works well and people really like it and Linux/MacOS get jealous, then we'll build something analogous on those systems.

The only constraints are:

• Windows installation from standalone .exe and .zip file will continue to work for the foreseeable future. That will always be the best-tested method from my perspective, since it's what will continue to be used by CI and such.
• Packages should avoid depending too heavily on aspects of installation. E.g. if we were to start bundling a small stand-alone Jupyter installation to ease IJulia installation on windows, I would want that to be done in such a way that it could auto-download whatever was bundled in the event that a user installs through a different mechanism.

Other than that, I see this kind of like another distribution packaging effort like the RPMs that nalimilan has built since forever. I say go wild, and if there are roadblocks in your way on our end, (like that codesigning issue) continue to make noise so that it stays on our radar and we can get stuff fixed for you.

[staticfloat]

@davidanthoff can you verify that the latest nightlies are being signed more thoroughly, and will work for your purposes here?

[staticfloat]

Ah, I think I should have posted that message here instead: https://github.com/JuliaLang/julia/issues/25730

[davidanthoff]

I'm under water with teaching stuff right now. I've put it onto my todo list :)

[davidanthoff]

Alright, I won't have much time for actual coding on this until the end of the year, but the next steps are all logistics in any case, and I think we can try to move that forward. I think we should try to just get this into the store next because all of the problems we've still had were related to the non-store setup story, and I think those should go mostly away once this ships via the store. Only way to find that out, of course, is to actually put it into the store.

Here are the steps we need to do next:

• [ ] Get NumFocus in on this, when we last chatted about the plan we said they would be the "publisher" of Julia in the apps store.
• [ ] NumFocus will need to get a code signing certificate in their name and I need access to that so that I can sign things.
• [ ] NumFocus will have to create MS Partner Account and sign up as a app publisher for the Windows Store in their name.
• [ ] I need access to the MS Partner Account for app publishing.

@aviks, could you maybe introduce me to the right NumFocus folks to discuss this?

[musm]

Why does NumFocus have to get involved? We can just create a Julia publisher and manage that ourselves. I don't see why their sponsorship would warrant them to be the publisher?

[davidanthoff]

We need an official, legal entity that is the publisher. Both the code signing authority and the MS store will check whether the entity we submit exists in a legal sense. So if we wanted to submit this as "Julialang", we would probably have to start a non-profit organization, get the tax exempt status and all of that. That is a lot of work, and when @aviks and I chatted about this last time we came to the conclusion that NumFocus would be the much easier route here, if they are on board.

[musm]

There's so Julia Computing as an option?

[davidanthoff]

Yes, but when I talked with @aviks about this we both felt that it would not be appropriate for a private company like Julia Computing to "own" the store version of Julia.

[Keno]

Julia Computing is already the code signing entity for all Julia releases on all platforms and has been for years. I don't have anything against the NumFocus plan, but that just seems like extra effort when it sounds like all we have to do is for somebody to sign up for the MS account.