Julia: Move SuiteSparse bindings to a package

[ararslan]

Note that SuiteSparse is only the set of solvers for sparse arrays, not the entire sparse array functionality, though a case could be made to remove that from Base as well.

[ararslan]

A challenge in doing this is setting up the resulting package to fetch the library. For most things we can just use BinDeps to get binaries from a system package manager if they're available, otherwise build from source. However, system package managers vary widely in their offering of SuiteSparse, and older versions can't build as a shared library by default, if I'm not mistaken. Further, WinRPM doesn't have a package for SuiteSparse, and typically the requisite C compiler for libraries like this isn't available by default on Windows. Thus I'm not sure what the best solution for this would be. Any suggestions would be welcome.

[jebej]

Wouldn't it be possible to compile with gcc from WinRPM?

[ararslan]

Could be, I was thinking about that as well.

[jebej]

Some packages have started manually calling gcc.exe to do that (eg Clipper.jl). It would be nice to integrate gcc into eg BinDeps to automatically handle compilation on Windows. At that point we will then be able to rely on there being a compiler on Windows.

[ararslan]

Though it may work in some cases, that doesn't really seem like a good general purpose solution. There are also licensing concerns with distributing GCC, since it's GPL.

It's likely that for SuiteSparse I'll end up using Travis and/or AppVeyor deployment to GitHub releases so that no local compilation will be necessary; the appropriate binaries will be installed along with the package.

[jebej]

Is that the reason for the thumbs down?

[ararslan]

¯\_(ツ)_/¯ It's hard to know others' concerns if no one voices them. As such, I can't speak for the others, distributing GCC with BinDeps (or some similar approach) just doesn't seem like the right general purpose solution to me.

[jebej]

I didn't mean to suggest to distribute gcc, but to use WinRPM gcc, which is what the aforementioned Clipper.jl is doing.

[jebej]

It seemed like a good solution since I see sometimes see people mentioning how Windows not having a compiler installed by default is limiting.

[ararslan]

I'm not a BinDeps expert so I can't really speak to all of he pros and cons of various approaches, but it's not a _bad_ suggestion in my opinion, and I do think it warrants further discussion. I recommend opening a discussion on the matter on Discourse so that hopefully others can weigh in without this issue getting too off topic. 🙂

[lobingera]

@jebej I voted with 'down' in relation to some story some time ago when it was decided that 'julia is not a distribution system for git'. The same would apply for 'BinDeps is not a distribution system for a c-compiler'. BinDeps has exactly the one job to support binary dependencies mostly in the form of shared libraries that can be accessed by ccall. I can see the need of a general BuildSharedLibrary.jl that can build from sources a library - maybe included into package that are closely connected to a compiler (like Clang.jl and Cxx.jl). So i voted against including a build system into BinDeps, but not against providing a build system.

[tkelman]

WinRPM can give you a compiler executable but it can't give you an environment that can run configure scripts or makefiles.

My personal plan is to split and replace BinDeps with two pieces, a BinaryBuilder.jl that makes it easy for package authors / maintainers to automatically build portable binaries across the major platforms (then once up and running, make rebuilds and version updates simple to automate on CI and upload to github releases), and a simple-as-possible BinaryDownloader.jl for users at install time that just downloads and extracts these tarballs. I'm working on writing up the design and plan in a Julep and putting together the complicated pieces. It should look somewhat like a decentralized version of conda-forge (without the conda). This obviously isn't ready yet though, so for now it's best to get the shared library build going from source on Linux, and prepare a dll download for Windows. Those dll binaries can be cross-compiled by the package maintainer if you don't have access to Windows.

[lobingera]

@tkelman

I'm working on writing up the design and plan in a Julep

Is this visible already, i didn't find it?

[tkelman]

Not yet. A bunch of notes and thoughts and a few scripts, very disorganized still.

[jebej]

Thanks for the explanations. It seems we all were suggesting pretty much the same thing (BuildSharedLibrary.jl or BinaryBuilder.jl). I had proposed using WinRPM's gcc in BinDeps.jl since its purpose reads "Tool for building binary dependencies for Julia modules".

[ararslan]

@ViralBShah moved this to the stdlib, so this is effectively done. We can move it out of the repo entirely later.

[tkelman]

That should be tracked, given it's the last dynamically loaded GPL dependency without a linking exception

[ViralBShah]

Given that all of stdlib will need to be moved out anyways, I think this will happen. Needs BinDeps2. I guess we could leave it open but move it off milestone.