Julia: Deploy pre-built dependencies onto CI

[staticfloat]

Now that BinaryBuilder.jl is picking up steam, I think we can simplify our CI infrastructure a little bit by building dependencies such as LLVM, OpenBLAS, etc... offline for all the systems we care about, then at CI time, just downloading the relevant tarballs, extracting them into usr and continuing the build. I think using prebuilt binaries for these large chunks will significantly simplify our build process, and should help make CI more reliable and easier to iterate on.

The deps we could prebuild (and their status if someone's already started building it) are:

• [x] ARPACK (Done: https://github.com/JuliaLinearAlgebra/ArpackBuilder)
• [x] OpenBLAS (Done: https://github.com/JuliaLinearAlgebra/OpenBLASBuilder)
• [x] curl (Done: https://github.com/JuliaPackaging/Yggdrasil/tree/master/LibCURL)
• [x] dsfmt (https://github.com/JuliaMath/DSFMTBuilder)
• [x] GMP (https://github.com/JuliaMath/GMPBuilder)
• [x] libgit2 (Done: https://github.com/JuliaPackaging/Yggdrasil/tree/master/LibGit2)
• [x] libssh2 (Done: https://github.com/JuliaPackaging/Yggdrasil/tree/master/LibSSH2)
• [x] libuv (Done: https://github.com/JuliaPackaging/Yggdrasil/tree/master/LibUV)
• [x] LLVM (Done: https://github.com/staticfloat/LLVMBuilder/)
• [x] MbedTLS (Done: https://github.com/JuliaWeb/MbedTLSBuilder)
• [x] MPFR (https://github.com/JuliaMath/MPFRBuilder)
• [x] Openlibm (Done: https://github.com/JuliaMath/OpenlibmBuilder)
• [x] PCRE (Done: https://github.com/staticfloat/PcreBuilder)
• [x] SuiteSparse (Done: https://github.com/JuliaLinearAlgebra/SuiteSparseBuilder)
• [x] libunwind
• [x] libutf8proc

As I continue to build pieces of the ecosystem, I'll update this list.

[StefanKarpinski]

This can be done piecemeal, right? I.e. use prebuilt binaries for the things where they exist while we continue to build dependencies from source for the rest.

[ararslan]

BinaryBuilder will need https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/32 before we can use it with our full CI infrastructure here.

[ViralBShah]

I updated the list above. If this is the way forward, I can put together a few more of the Base dependencies.

[ViralBShah]

Also, perhaps for some of the simpler dependencies, we should start using BinaryBuilder already rather than waiting for everything.

[StefanKarpinski]

Yes, I don't understand why we would wait for everything here. It seems far better to start using BinaryBuilder for the dependencies for which it's available. We also do __NOT__ have to wait for FreeBSD to do this: we can use BinaryBuilder on some platforms and not others.

[ViralBShah]

Not using it everywhere only doubles the workload though.

[StefanKarpinski]

Workload on what?

[ViralBShah]

Having to maintain a build process on BinaryBuilder and having a different build process for FreeBSD (and other unsupported platforms) is twice the workload for the people maintaining the build. I think what may be acceptable is not stop the move to BinaryBuilder for all supported platforms now, and have other platforms catch up later.

[StefanKarpinski]

We already have working builds for all of these things, it's not like we have to develop them. If one of them stops working at some point then someone can choose to either fix it or just do the work to switch fully to using a builder. If we wait until we have all of the builds for all of the things to use any of them then we will never switch and we will never work out the bugs.

[ViralBShah]

Sure. That makes sense.

[ViralBShah]

@staticfloat Are you ok to move OpenBLASBuilder to JuliaLinearAlgebra. Perhaps OpenLibmBuilder to JuliaMath?

[staticfloat]

I need access to JuliaMath to move OpenlibmBuilder

[ararslan]

I've added you.

[ViralBShah]

Getting close to having full BinaryBuilder coverage on Base libraries (even though Arpack and SuiteSparse need to move out, we have them ready now). The produced binaries need to be tested and such.

[ViralBShah]

Should we move LLVMBuilder to JuliaLang? Where should PcreBuilder and all the curl, libgit2, unwind, mbedtls etc. go?

[ViralBShah]

Is there an existing PR on how to enable Base to start using libraries from BinaryBuilder? Is LLVM the right one to look at? It will be great to use the same binaries for all developers, for releases, and reduce compile time overall.

I assume the process should be to basically just download the relevant tarball and untar it. Is there anything more to it than that?

[ararslan]

Builders for curl, libgit2, and MbedTLS seem like they would be at home in the JuliaWeb organization.

[ViralBShah]

@quinnj Would you be ok moving MBedTLSBuilder to JuliaWeb? I am happy to put the rest of these into BinaryBuilder form. Can someone give me owner access to JuliaWeb?

[ViralBShah]

I guess libuv and libunwind should be in JuliaLang. What about utf8proc? JuliaStrings?

[ararslan]

Since utf8proc lives in JuliaLang, a builder for it living here would make sense to me.

I'll add you to JuliaWeb. Edit: Never mind, I can't. (Member, not owner.)

[StefanKarpinski]

We can move utf8proc to the JuliaStrings org. We could ask @nolta if he'd mind transferring ICU.jl to JuliaStrings and if @nalimilan is willing we could move StringEncodings.jl there too.

[StefanKarpinski]

Also: dang, some progress has been made on this list.

[quinnj]

MbedTLSBuilder is already in JuliaWeb............https://github.com/JuliaWeb/MbedTLSBuilder

[ViralBShah]

Ah - it was just in the list above that we had the old URL. Updated it.

[ViralBShah]

Perhaps LLVMBuilder should also go to JuliaLang?

[nalimilan]

We can move utf8proc to the JuliaStrings org. We could ask @nolta if he'd mind transferring ICU.jl to JuliaStrings and if @nalimilan is willing we could move StringEncodings.jl there too.

Fine with me for StringEncodings.jl. But there are currently two different repos for ICU.jl, and the one in JuliaString org includes a few significant fixes/improvements.

EDIT: I've just transferred StringEncodings.

[StefanKarpinski]

I transferred utf8proc as well.

[ViralBShah]

Closing this with #31441