Julia: Shrinking Base and Introducing a Standard Library

[tknopp]

In #4898 there is a discussion whether the reduced loading time of the wonderful static compilation work should be an argument to either shrink or expand the Base module.

As it seems to be difficult in general if things can go in to Base or go into a package (i.e. where to draw the line) I would like to propose the following:

• Shrink Base so that the initial load time and memory consumption is even better
• Introduce 4-5 modules forming the "standard library" of Julia.

Whether these standard modules should be automatically loaded is from my perspective only a minor discussion point. More important is that this behavior can be easily changed (e.g. in juliarc)

One concern might be that one cannot rely on what module has been loaded in the users environment. But this can be solved by simply always explicitly importing the standard libs when developing a package. "using StdLib" could be a shortcut to import all standard libraries.

It might make sense to offload the standard modules into packages to make the development flow easier. One could then pull in the standard lib when building Julia.

I have not a concrete proposal for standard modules but "LinAlg" and "Signal" (like scipy.signal) come to my mind.

[gitfoxi]

To paraphrase myself from #4898, this seems like a case of premature optimization. No need to ape patterns from languages developed in the covered-wagon era of computing. If anything, look in the other direction. Assume a super-grid cloud cluster of overclocked n-way super-scalar VLIW stacked-die SMP GPUs.

Performance is quite a separate issue from namespace pollution. On that topic, more thought is required -- or possibly less. Is anyone complaining about the namespace yet? Might as well leave it alone until someone does.

[JeffBezanson]

@gitfoxi makes a good point --- being raised on OOP, we all reflexively see more modularity as a good thing, but we have to be clear what exactly we are trying to solve or obtain. It would be nice to have a "minimal julia" perhaps for constrained environments. But I think we can get these sorts of things without changing the default user experience.

see also #1906, #3128.

[tknopp]

I should have been clearer of the purpose of this proposal. Actually the main point is to have "privileged" packages (#1906) which are shipped with Julia. Then it might be easier to draw a clear line between what is in Base and what is in the Standard library. The "minimal Julia" mode would naturally evolve out of this reorganization but it is not the main driver for my proposal.

Maybe I was just surprised to see things like fft and sparse matrices in Base but no spatial filtering (gauss filter and so on)

If this proposal goes into the wrong direction, feel free to close it.

[StefanKarpinski]

I don't actually think that doing using LinAlg at the top of a file before using linear algebra stuff is so bad. That's explicit and easy enough. My main objection to the way that "object-oriented" language like Python and Java do things is that they make the things you have to import so granular that every time you reach for new functionality you need to add imports. But we're not talking about anything like that. This would be one or two using lines at most.

[pao]

It would be nice to have a "minimal julia" perhaps for constrained environments. But I think we can get these sorts of things without changing the default user experience.

As someone who gets to do embedded work, I think that's the right approach. When I know I'm doing a final deployment of a static codebase to a target system, give me the opportunity to strip that down, but don't make things (much) harder on the people who are actually writing the algorithms. The popularity of Simulink Coder/MATLAB Coder/Embedded Coder come from being able to have a full development environment on the one hand and deploy something much more basic on the other. Being able to do those things in a single language without a translation step is one of my hopes for Julia.

[tknopp]

It seems that I am not good in expressing my thoughts. I was proposing that Base+Standard should have more functionality than currently in Base. And these could be loaded by default. This means that the "default user experience" is improved instead of degraded. And on top of this, for people doing embedding stuff, or simply people that want to program smartphone apps in Julia there would be an easy way to remove stuff that is not needed.

For me, Julia is not the better Matlab but actually one of the most promising alternative to programming in C++. Julia solves the "template issues" of C++ in such a beautiful way that it is hard to get back to C++ for the day work.

[gitfoxi]

If there's one thing we can all agree on, it's that C++ can go straight to hell.

[StefanKarpinski]

It's been a bit surprising but in retrospect makes sense that Julia is a pretty good C++ replacement. Both languages allow you more abstraction than C with similar performance, letting you consciously trade-off higher-level programming styles for efficiency, all the way down to writing C-like code and getting C-like performance (and frequently getting C-like performance without having to give up high-level style). Julia's parametric types and "everything is a template" approach has a very similar effect to C++ templates, but with far less brittleness and hassle. And of course, C++ uses operator overloading rampantly and Julia's operators, being simply generic functions with special syntax, give you operator overloading as it should be. So if the main things you liked about C++ were:

1. performance
2. generic programming
3. operator and method overloading

then Julia is a phenomenal C++ replacement. If, on the other hand, the things you need from C++ extend to manual memory management, pointer arithmetic, and multithreading – which are very legitimate needs for some applications – then Julia is not such a great C++ replacement. But I think there are a lot of C++ programmers for whom the first three are really important and the last three are not.

[JeffBezanson]

Very tentatively added to 0.4. This may not be as high-priority as other stuff though.

[timholy]

If it is done _after_ really good package precompilation, then it should be the second highest priority item for 0.4 (right after Keno's debugger) :smile:

[tknopp]

@JeffBezanson Is there some general general platform (github issue or julia-dev mailing list) to discuss what should be targeted for 0.4?

@timholy Just wanted to add a comment on this but your message was faster :-)

My take: 0.4 has already various things that are "almost done" and if one targets 6 month for developing 0.4 I am not so sure if package precompilation and refactoring is doable. Esspecially as package precompilation can only be done by a limited number of people (Jameson, Keno, Jeff, ...).

In short: I am a big fan of package precompilation and the shrinking base. But please lets not add things to 0.4-projects which "may" happen. Better merge all the open important issues from the last half year, stabilize and release.

[timholy]

@tknopp, look for issues with the 0.4-projects tag. Here's a link: https://github.com/JuliaLang/julia/milestones/0.4-projects

[tknopp]

Thanks Tim. Thats clear. But the question was more about whether there is a central place to discuss what should get the 0.4-projects tag.

[ivarne]

Isn't this issue mainly about moving things out of the Base module and into separate modules that can be imported separately with a using ... statement?

The job here seems to be that these modules must have explicit (and few) dependencies. Some sort of standard way of importing everything might also be good for interactive usage. Whether they are precompiled into the default sysimg, or lazy loaded from code or a dll is a totally different issue.

[tknopp]

@ivarne Yes exactly. Please have a look at my related comment https://github.com/JuliaLang/julia/issues/7926#issuecomment-51897732

Technically it might make even more sense to put the modules into separate packages ("default packages") that are pulled when running make.

[timholy]

Precompilation would be fine, but parse/compile from source code would be a massive step backwards. Some of us already have to wait ~1minute before Julia starts doing anything useful, and this would make it worse.

[ivarne]

This issue is not about making startup slower. It is about changing the structure of Base into less interdependent modules. All the new modules can (and should) be compiled with Julia as they do today.

[tknopp]

Yes although there might be different technical solutions. If we would have package precompilation it would be easier than without. One strategy would be:

Step 1: Move things into independent modules within base.
Step 2: Move the modules into individual packages that are automatically pulled during make (i.e. git submodules)
Step 3: Use precompiled packages.

One motivation of this proposal is that it hopefully lead to a much larger standard library. Image.jl is one examplary candidate. I can't live without gaussian multidimensional filtering in the Julia "standard library"...
More controversial might be to pull ploting and a GUI toolkit into the "standard library" but in my opinion this is would be a great way to move forward.

[quinnj]

I actually have a branch that allows a Make.user variable of NOBIGFLOAT that will exclude all BigFloat functionality. Some thoughts having gone through that process:

• I think #6195 will help with this. The problem is that even with a minimal dependency like BigFloats that is mostly self-contained, there are little snippets that have sprouted up all over Base, which are quite awkward to just move into the MPFR module (constants.jl, printf.jl, etc.). The other option (which I've done in my branch) is to have a @BIGFLOAT macro that acts like @windows_only code.
• Even with conditional modules, some dependencies like libuv and PRCE are literally _everywhere_. Those would take major surgery to remove or make optional.
• That said, I think this is all worth the effort, though it would take a concerted effort.

[StefanKarpinski]

Moving functionality into packages and precompilation are completely orthogonal. We happen to create a system image before loading any external packages right now, but there's no reason that has to be the case. We can just as easily load Base, load various default packages, and then generate a system image.

[StefanKarpinski]

Opened an issue just for precompilation: https://github.com/JuliaLang/julia/issues/7977.

[lucasb-eyer]

Besides embedded, I'd like to add another use-case for switching off features with dependencies: using Julia as an extension-language in a larger (C/C++) project. Julia is the first language of which I believe it can become a serious alternative to Lua. We're not there yet, though.

[tknopp]

@lucasb-eyer: Jep, this was one of the primary motivation at the point I proposed this. And actually I think that Julia will involve in this way. But it is more a long term goal which will need a joint effort. In the 0.4 phase there are to much other important things to happen so that I would keep this on hold after the 0.4 (or even 0.5) release.

[mikewl]

I was reading through this and #1692 and noticed this was not directly addressed.

Would the packages be able to be updated? I realise that there would probably have to be some branches to deal with breaking previous versions of Julia but I think that one of the strengths of having separate packages is that bug fixes and improvements could be had without needing to update Julia.

[jiahao]

See #10333 (comment) and the discussion that followed.

Summary: Pulling modules out of Base is really hard to do without major breakage, because of the way different versions of the same package export identifiers (particularly types and module names) into the current namespace. The situation is compounded by Julia (currently) disallowing the redefinition of types and the current scoping rules precluding conditional redefinition of existing identifiers with runtime if statements (see JuliaLang/Graphics.jl#1 (comment)).

[mikewl]

Sorry, I was thinking post-removal.

I was reading https://github.com/JuliaLang/julia/issues/5155#issuecomment-51922845 and wondered if doing this would make updating those packages significantly more difficult than if they were precompiled separately. Say if StatsBase was a part of the Standard Library and after installing Julia you wanted a newer version than the one included with the install (assuming one is available).

[hayd]

Could the beginnings of this be reorganising the base directory? atm only some parts have their own directory (grisu, dates, markdown, ). IMO grouping files into subdirectories (maybe numbers, arrays, strings, terminal, ...) would make it easier to understand how julia code is arranged/works and make it clearer what can/should be pulled out into their own modules (or potentially, as some other issue talked about, be hidden behind an import but still in the Base installation).

[ViralBShah]

This has only be done piecemeal - largely based on the personal tastes of people working on those pieces.

[tknopp]

@jiahao: I think we have to distinguish two things here: Removing stuff from base that belongs into a non-default package and removing stuff that belongs into a default package. For the later be still miss the infrastructure to realize this.

That things break it IMHO not really an issue. We are on the road to 1.0 and as long as the breakages happen at defined points (0.3 -> 0.4) thats ok. The problem is that we try to have packages that work on two versions (0.3 and 0.4) with that assumption in mind we will never realize such restructuring as needed to shrink base and introduce a standard library.

In short: This issue is hard, dirty work but if we have solved it everything will become more maintainable.

[StefanKarpinski]

It's somewhat hard to modularize the base code since it ends up having to be defined in a fairly particular order for bootstrapping purposes. I do think we should work more towards that, but it's kind of tedious and somewhat thankless work, since in the end you have the same functionality, just organized differently.

[tkelman]

I do think we should work more towards that, but it's kind of tedious and somewhat thankless work, since in the end you have the same functionality, just organized differently.

The module precompilation work however could result in a major payoff for this kind of reorganization - if modules can be bootstrapped individually and made into separate shared libraries, we could specify the interdependencies in the makefiles and parallelize the bootstrap process. This would improve turnaround time for changes to base for everyone.

[ivarne]

We could also possibly have a make small target that don't precompile the optional modules, saving even more time.

[tkelman]

Quoting @tknopp from https://github.com/JuliaLang/julia/pull/6193#issuecomment-106827239

Thanks @tkelman this is exactly the important point that I also tried to make clear in our SparseVector discussion. I know, the case is a little different and don't want to processed the discussion here, I just want to highlight your sentence and its importance for any future discussion on base inclusion (and also #5155)

The two discussions are related, but I didn't want to cross them too closely, since there are some important differences. The existing FFTW bindings are working fine, and the new implementation does not currently exist as a package. Whereas SparseVectors do exist as a package, but there are bugs and semantic inconsistencies in the current base implementation and indexing of SparseMatrixCSC, which bringing SparseVectors into base would fix. Long term, neither sparse matrices (or vectors) nor FFT's should need to be included in the Julia runtime system image for a compiled standalone hello world script written in Julia. Though some generality and accounting for sparsity will need to make its way in some form into various array operations that would always stay in base (indexing, reductions, broadcast, etc), I don't think FFT's have quite the same requirements - but I could easily be wrong.

[ScottPJones]

If anybody wants my help, I'll put my time where my mouth is to try to make "Julia Lite" a reality... (at least, as soon as we ship first version of our product! :grinning:)

[tkelman]

One action item that needs some research work is determining the best way of bundling "default packages" along with the Windows and OSX installers, and in the generic Linux tarballs. We could put them into share/julia/site/v0.x or something (which appears to already be included in LOAD_PATH by default) and easily build separate "lite" and "full" binaries, but we'd need to experiment a bit with how well putting packages there interacts long-term with Pkg.update() usage, would you end up having separate copies in vs outside of Pkg.dir, etc.

[Mx-Glitter]

I'm largely in favor of such a modularization, if you want a concrete example (IMHO) done right, you can look at the (trigger warning: PHP ahead) Symfony Framework with its Standard Edition: core Symfony is a collection of "components" and core "bundles", i.e. base modules in your example, and Symfony Standard Edition is core Symfony + standard "bundles", i.e. standard modules.

(Difference between components and bundles is that bundles are to be used directly in the framework, it's the equivalent of Julia packages whereas components are standalone libraries).

The decoupling even allow you to use any component separately from a Symfony project, e.g. if you only care for a Yaml parser and not a whole web app framework.

[vtjnash]

with #8745 merged, I figured I should add a little demo here of some relevant functionality that is now available:

$ git diff mmap.jl
diff --git a/base/mmap.jl b/base/mmap.jl
index b1de73d..41048fa 100644
--- a/base/mmap.jl
+++ b/base/mmap.jl
@@ -1,6 +1,7 @@
 # This file is a part of Julia. License is MIT: http://julialang.org/license

-module Mmap
+eval(Base, :(
+module Mmap2

 const PAGESIZE = Int(@unix ? ccall(:jl_getpagesize, Clong, ()) : ccall(:jl_getallocationgranularity, Clong, ()))

@@ -208,3 +209,4 @@ end
 sync!(B::BitArray, flags::Integer=MS_SYNC) = sync!(B.chunks, flags)

 end # module
+)) # quote eval

$ cd base

$ ../usr/bin/julia --output-incremental=yes --output-ji mmap2.ji -J ../usr/lib/julia/sys.dylib mmap.jl 
<nuisance warning clipped>

$ ../julia -q
julia> Base._require_from_serialized(#=node=#1, "./mmap2.ji", #=toplevel=#false)
1-element Array{Any,1}:
 Base.Mmap2

julia> using Base.Mmap2

julia> Mmap2.<tab>
Anonymous            MAP_ANONYMOUS         MS_ASYNC              PAGESIZE              eval                  mmap
F_GETFL              MAP_PRIVATE           MS_INVALIDATE         PROT_READ             gethandle             settings
INVALID_HANDLE_VALUE MAP_SHARED            MS_SYNC               PROT_WRITE            grow!                 sync!
julia> Mmap2.mmap
mmap (generic function with 24 methods)

[timholy]

Real-time surgery---pretty fancy! Can you clarify whether this is significantly different from include("base/mmap.jl")?

[vtjnash]

that example is to include as compile is to require, e.g. they are effectively the same it's just a matter of when the 'surgery' happens

[mikewl]

For the windows and osx installers, couldn't the package path be set via the installer? So default will remain as is or it could be changed and this will then set the environment variable correctly?

Then there will be no disparity between the standard library and any other packages. The standard library just consists of packages deemed appropriate to include in the Julia install. Pkg.update() would then update the standard packages as well as any user included packages.

The generic Linux tarballs I have not used so I can't comment on them.

[tknopp]

I think it would be clever to move the standard library to a real package that can be updated via Pkg.update(). The question is if there is one StdLib package or several smaller packages, which composed are the standard library.

[mikewl]

A built in reexport would be nice for this kind of purpose. Several smaller packages could make up StdLib. Then you could either just have using StdLib for full functionality or pick and choose the appropriate packages as required. using StdLib would then re-export all exported items from the associated packages. So StdLib can then easily be modified to include more packages if so desired.

I would think smaller packages would be more ideal for this kind of purpose. Sparse functionality would not be required by most or FFTs but they could make a good case for being part of the StdLib.

Some bikeshedding would probably have to occur as well over the most appropriate name for this kind of thing.

[JeffBezanson]

I'm just going to start making a little list of pieces that would be good to remove from Base. No commitment yet as to how these should be handled --- some may be packages, some may be something else.

• Dates
• Profile
• LibGit2
• Pkg
• FastMath
• QuadGK
• DSP
• FFTW
• SparseMatrix
• LinAlg
• Markdown
• Terminals
• LineEdit
• REPL
• Enums (?)
• DataFmt (possibly merge with DataFrames or similar)
• Mmap (not large, but heavily OS-dependent)
• Poll
• Multi (and managers.jl)
• Channels
• Combinatorics
• Random
• GMP, MPFR (however, requires defining the types and basic ops in Base)
• Primes
• SpecFun
• SharedArrays

[simonbyrne]

Why would the types and basic ops would need to be defined in Base for GMP and MPFR?

[JeffBezanson]

It's maybe not 100% required, but a lot of code in Base uses BigInts. It's also part of the syntax, needed for parsing numeric literals.

[simonbyrne]

Ah, of course.

One other candidate might be Float16, though this may depend on future GPU functionality where it is actually useful (depending on whether the GPU stuff is in a package or Base).

[tkelman]

It's also part of the syntax, needed for parsing numeric literals.

Does it really have to stay that way though?

[JeffBezanson]

Thought about Float16, but the code for it is so small it might as well stay.

Does it really have to stay that way though?

I think I'd prefer it. It doesn't sit well to be unable to simply write a big integer in a program. Plus, the grisu code already contains a bigint implementation --- even if we can't reuse it, it makes it hard to argue that such functionality is not allowed in Base.

[tkelman]

there are lots of types we don't have literals for, and bigint is a pretty hefty dependency to have in the front end for parsing. we don't allow silent promotion to bigint when doing arithmetic on integers, but we do when parsing integer literals. feels inconsistent.

[jakebolewski]

Pretty much all high-level languages dynamic languages have BigInt support though.

[johnmyleswhite]

I agree with Tony: the fact that adding a single digit to a literal can change its types seems a little weird -- especially in a language for which enforcing consistent types is so much important than in other dynamic languages. It's potentially a surprising source of type variability if doing something like the following changes the type of x:

julia> x = 111111111111111111111
111111111111111111111

julia> x = round(Int, sin(x))
1

[jakebolewski]

@johnmyleswhite are you advocating that all non floating point literals are parsed as Int? Otherwise you are also giving up Int128 support for signed literals. Unsigned literals are lexed as into the smallest bounding literal type.

I guess what I'm saying is the problem you raise is not limited to BitInt

[johnmyleswhite]

What I would advocate for, but won't argue hard for if no one else agrees, is that you only get Int128 or BigInt if you use a suffix.

[jakebolewski]

If we are able to shrink Base down to only requiring MPFR and GMP that would be amazing.

I'm with Jeff, I don't see this as such a big problem. We should have a better way of disabling these dependencies if people don't want to rely on them (no GPL, reject programs during lexing) but I don't think that should be the default.

[jakebolewski]

The larger question here is how to do this without massive breakage. Moving a couple of the dependencies out of Base did not go well last time and it is hard to do without breaking anything. I think this may be a rip the bandaid off situation but we might not have much leverage to do this going forward.

[JeffBezanson]

BigInt literals don't require GMP. We already have a bigint implementation in julia in the grisu code, and it doesn't require a crazy amount of code. Ideally, most of BigInt and its functions would be defined in julia, and we'd only (optionally) call GMP for better performance and advanced algorithms.

A couple items on the list are obvious candidates for packages, or for inclusion in existing packages. For the others, I would move them out of the base directory but keep them in this repo. Then we set things up so that you can load them with using, and somehow make it the default at the REPL to include all of them.

[tkelman]

I think #11638 was a good start in naming and organization. The obvious GPL libraries, SuiteSparse, FFTW, and Rmath will need to be moved out and handled via BinDeps. Package bundling also needs to be experimented with to maintain a full-featured "computing environment" distribution on top of a smaller mandatory core - some of this needs restructuring work within Pkg. That work needs to be done no matter what and can start before doing anything in base. The rest of the code reorganization should probably be done in place to start with, to first allow optionally disabling large chunks of Base and see what happens.

At least with hex literals what you get is easily predictable from the length of what you type, with signed literals it's more difficult to parse at a glance. I'd rather eventually have it be explicitly annotated and be a library feature, not a deeply entrenched part of core parsing in the "language spec." It's not something that needs to be dealt with immediately, but long-term by 1.0 it's worth thinking carefully about. Do we really want the length of integer literals to determine whether code can be deployed to an embedded environment where GMP doesn't run? It's not a very portable library to get built in exotic environments, and making it more optional and replaceable with alternative bigint libraries (or none at all!) for different use cases would be a good thing IMO. GMP has been a definite hurdle to doing standalone deployment of Haskell binaries, as a closely related example here.

Another entry to the list, heavily depended on but worth isolating in its own module, is Regex/PCRE.

[mikewl]

For automatically loading certain packages, would a repl init script which runs some Julia code automatically not work? The nice part would be then that it could be edited by the user after installation to include other packages that user would commonly use.

[JeffBezanson]

My thinking is that the only reasonable interpretation of a long digit string is a BigInt, and the only reasonable syntax for a BigInt is a long digit string. I also suspect that most uses of BigInts don't require GMP. We can get parsing, conversions, comparisons, and arithmetic with small routines in Base written in julia.

[GunnarFarneback]

Is Rational needed in Base?

[StefanKarpinski]

Rational is useful for expression constant coefficients in a way that allows generic programming.

[simonbyrne]

2025 probably needs to be addressed before we can pursue this _en masse_. For example, a library may provide a special implementation for a BigFloat, but be able to operate without them (where would besselj0(::BigFloat) be defined in the above layout?).

[timholy]

Also, we can't currently precompile methods that span modules, so breaking things up (a goal I support overall) will have some downsides in terms of loading/JITting time.

[KristofferC]

Is this still planned for 0.5.0? It seems that it would require a lot of work that might be more useful at other places?

[wildart]

I think that package management system (Pkg) is a good candidate. We already cut the development related part from it. I see no reason to stop at this point. Considering that Pkg introduced additional dependencies to the main project (libgit2, libcurl, and possible candidate - libmbedtls), I think it would be beneficial to spin it as separate project. It will give more room to work on new features (namesapces, GUI), and the project itself wouldn't be constraint by Julia release plans or other limitations, e.g. licensing issues.

[StefanKarpinski]

If the package manager is a package, how do you install it?

[StefanKarpinski]

Although I definitely agree that it makes sense to be able to strip it out easily.

[wildart]

If the package manager is a package, how do you install it?

Install it as a system package, but separately. In case if PM is not installed, give an instruction how to install it or provide information on setup of a location from where packages can be loaded.

[tkelman]

Install it as a system package, but separately.

And for systems that don't have package managers? Technically separating out Pkg should be feasible, and building Julia without it may be something a small handful of people want to do for embedding or other use cases, but there's a lot of work to do in figuring out exactly how this would be distributed alongside Julia for the majority default configuration that will need it.

[tknopp]

And for systems that don't have package managers? Technically separating out Pkg should be feasible, and building Julia without it may be something a small handful of people want to do for embedding or other use cases, but there's a lot of work to do in figuring out exactly how this would be distributed alongside Julia for the majority default configuration that will need it.

That is actually the point of this issue: Building the infrastructure such that some "defaultPackages" file is parsed when building Julia. It could then automatically pull the default packages.

Once such an infrastructure is available one could think about moving stuff into packages. IMHO Pkgdoes not seems to be the most important package to split of.

I want to note that one has to distinguish between whats included in the julia repo and what is distributed when downloading julia. Till now this is the same but once we have some infrastructure for default packages this will differ.

[wildart]

Building the infrastructure such that some "defaultPackages" file is parsed when building Julia. It could then automatically pull the default packages.

Agree, it does not relate to ordinary user experience. Some parts of julia are already pulled from git repositories during build. It shouldn't be a problem to copy a fresh copy of the repository into a particular directory after the build process is done.

[timholy]

I've thought about moving Profile out, and am happy to do the work esp. if it motivates others to begin on their pieces, but Profile requires a few profile-specific ccalls (platform-specific signal handling). Before we begin migration, it would be nice to have a strategy for providing pre-built binaries not just of (trimmed-down) Julia but also "common" packages.

https://github.com/JuliaLang/julia/issues/5155#issuecomment-106843678

[tkelman]

Note that the migrated JuliaBox is now demonstrating some of the ways in which we're pre-bundling packages along with Julia. There have been some precompilation issues but we're working on fixing those.

[tknopp]

Question: Would it be required that the bundling happens during distribution of julia? What about doing this at the first execution of julia: I.e. ask the user: "do you want the minimal version or should we install some batteries?" Alternatively the minimal version could have a hint in the banner, what to execute so install the full version.

[vchuravy]

Also an important side note is that is would be nice to export the git history if possible to the package as well.

@andreasnoack and I have been discussing this for the sparse code.

[simonbyrne]

See https://github.com/JuliaMath/Primes.jl/pull/12 for how it was done with Primes.jl

[tkelman]

Yes that's a good idea for code migrations, I did that with filter-branch for https://github.com/JuliaPackaging/Git.jl.

[vchuravy]

For sparse I am using https://gist.github.com/vchuravy/81de73d34ed2b24de3123f50bb3c5bbd with the results at https://github.com/vchuravy/ExperimentalSparse4

[tlnagy]

Before we begin migration, it would be nice to have a strategy for providing pre-built binaries not just of (trimmed-down) Julia but also "common" packages.

I'm also worried about moving too many things out of Base before an effective "batteries-included" version of Julia is developed. Already I'm having to import a large number of packages before I'm able to get working, one of the things I liked least about python.

Would it be required that the bundling happens during distribution of julia? What about doing this at the first execution of julia

The latter would be much preferable. Having one meta-package that loads up base technical libraries at run time: Combinatorics, Primes, FFTW, QuadGK, DSP etc would be ideal. The user wouldn't see this after installing it (i.e. no using TechnicalJulia every time we fire up the repo).

[JeffBezanson]

Next items that should be easy to move: SharedArrays and Poll (file watching).

[JeffBezanson]

Closing in favor of #18795 and more specific issues for pieces to remove.