Julia: Update precompile.jl for 0.7 / 1.0

[KristofferC]

Creating this issue with tag so we won't forget.

[KristofferC]

Some specific stuff that I believe should be in here:

• Showing a help message (exercises some of the markdown code)
• Showing an error message (exercises stacktrace printing)
• Pkg3 status, Pkg3 add package, Pkg3 remove package, Pkg3 load package, very important for the snappiness feel of Pkg3
• Printing of vectors / matrices. Typically running something like rand(2,2) is one of the first thing you show someone in the REPL, nice to make that feel fast.
• Some general mucking around in the REPL

More stuff?

[KristofferC]

By moving stuff to the stdlib we need to slightly tweak the way the precompilation file is postprocessed. For example, the REPL methods need to go to stdlib/REPL/precompile.jl since REPL is not defined in Base.

[KristofferC]

Messed around a bit with this so requesting some feedback here.

Currently, I tried the following (one file) approach. Enable the tracing, run julia and output to precompile.txt and then run the following script on it:

const HEADER = """
# This file is a part of Julia. License is MIT: https://julialang.org/license

# Steps to regenerate this file:
# 1. Remove all `precompile` calls
# 2. Rebuild system image
# 3. Enable TRACE_COMPILE in options.h and rebuild
# 4. Run `./julia 2> precompiles.txt` and do various things.
# 5. Run `./julia contrib/fixup_precompile.jl precompiles.txt
"""

function fixup_precompile(precompile_file)
    precompile_file = joinpath(pwd(), ARGS[1])
    precompile_statements = Set{String}()
    for line in eachline(precompile_file)
        # filter out closures, which might have different generated names in different environments)
        contains(line, r"#[0-9]") && continue
        # Other stuff than precompile statements might have been written to STDERR
        startswith(line, "precompile(Tuple{") || continue
        # Ok, add the line
        push!(precompile_statements, line)
    end

    precompilefile_path = joinpath(Sys.BINDIR, "..", "..", "base", "precompile.jl")
    open(precompilefile_path, "w") do f
        println(f, HEADER)
        println(f, "module __precompile_area__")
        println(f, """
            if !(pkgid.name in ("Main", "Core", "Base"))
                @eval $(Symbol(mod)) = $mod
            end""")
        for statement in sort(collect(precompile_statements))
            println(f, statement)
        end
        println(f, "end # module")
    end
    println("Wrote a new precompile file to $(abspath(precompilefile_path))")
end

@assert length(ARGS) == 1
fixup_precompile(ARGS[1])

This writes a precompile.jl file that looks like:

module __precompile_area__
for (pkgid, mod) in Base.loaded_modules
    if !(pkgid.name in ("Main", "Core", "Base"))
        @eval $(Symbol(mod)) = $mod
    end
end
precompile(Tuple{Type{Expr}, Symbol, GlobalRef, Core.SSAValue, Core.SSAValue, Core.SSAValue, Core.SSAValue, Core.SSAValue, Core.SSAValue, Core.SSAValue, Core.SSAValue, Core.SSAValue, Core.SSAValue, Core.SlotNumber})
...

This just puts all the precompilation statements into one file. Since I "pseudo-import" all packages that are loaded, everything works nicely.
What is bad about this approach is that if you comment out a standard library that happens to have a method in the precompilation file, you will get an error there.

Possible solutions to this are:

• use the precompile file only as a starting point but filter away all methods that have a type that is in a non-loaded module at build time.
• assuming the stdlib dependency tree is a DAG, we can topsort it, and put compilation statements in the in the module which is the lowest in the graph. So a compilation statement involving both Pkg and REPL would go into stdlib/Pkg/src/precompile.jl since Pkg requires REPL. This is perhaps nice since the precompilation statment will follow along the package if it is moved out of the repo.

[quinnj]

At one point, @vtjnash said it wouldn't be a problem to include precompile statements for other modules; related?
https://github.com/JuliaLang/julia/issues/23548

[KristofferC]

I'm not sure how that is related, could you elaborate? If you want to run a precompile statement all the types in the statement need to be defined. So we can't put a precompile statement that includes Pkg in the REPL precompile function because we have no reference to Pkg there. We could put it in Pkg though, or in the base sysimg like I explained above.

[KristofferC]

For triage it would be good to discuss:

• Where the precompile statements should be. All in one file, or sorted into the stdlibs in some top-down order?
• What should be in the precompile? "Just go wild in the REPL" is perhaps underspecified. Should there be some script that presses keys, or something that is copy pasted into the REPL (together with some code movement to get the LineEdit stuff compiled)?

[JeffBezanson]

I think for expediency in 0.7 and 1.0 we can do the simplest thing and put it in one file. We can do something fancier later.