Julia: Precompile statement emitter sometimes emits invalid statements

[KristofferC]

This is OK:

julia> typeof(dateformat"YYYY-mm-dd\THH:MM:SS")
DateFormat{Symbol("YYYY-mm-dd\\THH:MM:SS"),

But running with trace-compile=stderr we can see the following precompile statement being generated

julia> Dates.format(now(), dateformat"YYYY-mm-dd\THH:MM:SS")
precompile(Tuple{typeof(Dates.format), Dates.DateTime, Dates.DateFormat{Symbol("YYYY-mm-dd\THH:MM:SS") ...

which contains an unescaped \ in the DateFormat type. Encountering such a type in precompilation gathering means it will fail.

[tehrengruber]

As already mentioned in #32574 I also stumbled upon this recently. As of now I am aware of three cases in which the output is not correct.

Case 1

Types with symbols as parameters which need to be escaped, e.g. contain a backslash (this is the case you mention)
_Example_:

foo(::T) where T = 1
struct A{B} end
foo(A{Symbol("a\\b")})

_Output_:

precompile(Tuple{typeof(Main.foo), Type{Main.A{Symbol("a\b")}}})

I have already fixed this case in https://github.com/tehrengruber/julia/commit/0fed122e57cbbd6e0bb3a7e6e56b1ab95420d10c (will open a pull request at some point)

Case 2

Types with symbols as parameters generated via gensym
_Example_:

macro add_parametric_type(expr)
  push!(expr.args[2].args, gensym())
  esc(expr)
end

@add_parametric_type struct A{} end

foo(::T) where T = 1

foo(A)

_Output_:

precompile(Tuple{typeof(Main.foo), Type{Main.A{363} where 363}})

Case 3

Types with a Char as a parameter, e.g. Char(0x56)
_Example_:

foo(::T) where T = 1
struct A{B} end
foo(A{Char(0x56)})

_Output_:

precompile(Tuple{typeof(Main.foo), Type{Main.A{Char(0x56000000)}}})

The origin of the problem is in jl_static_show_x_ here https://github.com/JuliaLang/julia/blob/faefe2ae64fb9a12824832582e2ddd3bc554d372/src/rtutils.c#L986 The bitstype Char (4 bytes) is traversed bytewise in reverse order, hence resulting in 0x56000000 instead of 0x00000056. I have no clue whats the reason for this ordering. @JeffBezanson you wrote this piece of code, any reason for the ordering that I'm overlooking here?

__Edit__

One thought further I recognized that the reason for the reverse byte ordering is endianess.

[pfitzseb]

We also sometimes generate code with invalid _s in it (in type parameters, IIRC).
See here for PackageCompilers workaround.

[rai-nhdaly]

Regarding Case 2, copying over this context from the original issue (https://github.com/JuliaLang/julia/issues/32574):

I would be willing to fix this myself, but I'm not sure how to handle this correctly. The generated symbol can not be used directly since the resulting code would be invalid just as well and replacing it by a different "regular" symbol, e.g. A{##363} where ##363 by A{B} where B, is problematic since B might collide with other parameters in general.

This is now addressable thanks to the cool var"" syntax that @c42f added in https://github.com/JuliaLang/julia/pull/32408.

So i think we just need to find whatever code is dumping the precompile statement (is it jl_static_show_x_?) and have it use the var"" syntax here instead! :)

This would be a very similar exercise to that taken in https://github.com/JuliaLang/julia/pull/34888.

[NHDaly]

I'm working on addressing Case 2, here: https://github.com/JuliaLang/julia/pull/38049

It's fixed on that branch, and now i'm just following up with a couple other small improvements to static_show() that I noticed along the way.

[NHDaly]

Via the logging added to PackageCompiler in https://github.com/JuliaLang/PackageCompiler.jl/pull/457, I've identified two more classes of failures. One of them might just need to be fixed in PackageCompiler, but I think the other one needs to be fixed in julia:

Case 4 - Incorrectly emitting precompile() with abstract types

UPDATE: Opened issue for this: https://github.com/JuliaLang/julia/issues/38149

For some reason, out of the 15627 statements emitted in our last run, we noticed 21 statements that returned false when running them in the output-o process. From what I can tell, these all appear to be cases where for whatever reason Julia emitted the precompile statement with abstract types, even though the function is _not_ marked @nospecialize. Sometimes they're ::Any, and sometimes they're other abstract types or type unions. This is not a valid precompile statement, since you cannot produce machine code for a function specialized to abstract types (at least as far as I understand).

Here are some examples we see:

┌ Debug: Precompilation failed: precompile(Tuple{typeof(Base.Broadcast.broadcasted), Function, Function, Array{MathOptInterface.ScalarAffineTerm{Float64}, 1}})
└ @ Main.anonymous none:33

┌ Debug: Precompilation failed: precompile(Tuple{typeof(Base.Docs.docstr), Any, Any})
└ @ Main.anonymous none:33



md5-e77e1215cf874148acdb410f3db0f5b4



```julia
┌ Debug: Precompilation failed: precompile(Tuple{typeof(DelveSDK._convert_input_value_type), Type{T} where T})
└ @ Main.anonymous none:33



md5-fb56b25580c89238a3da0618d084b658



```julia
┌ Debug: Precompilation failed: precompile(Tuple{typeof(Blobs.self_size), Type{Blobs.Blob{T}}} where T)
└ @ Main.anonymous none:33



md5-683940d9c96ec0615fbd47c92885dd55




I'm not sure yet why julia emits statements that will fail to precompile, but this is another case that we should try to fix.



### Case 5 - `UndefVarError` caused by unknown Module names.

UPDATE: Opened PR for this: https://github.com/JuliaLang/PackageCompiler.jl/pull/463

I _think_ this one is probably just an issue in PackageCompiler where we're not re-loading all of the modules from the user's program correctly?

In our latest run producing 15627 statements, we had 292 `UndefVarError`s that all appear to be related to module names not being available. Here are some examples:



md5-8fe6b58a66c33651a0ce8b2f4505d420



```julia
┌ Debug: Error executing precompile(Tuple{Type{Base.Pair{A, B} where B where A}, Symbol, Plots.Plot{Plots.GRBackend}}):
│ LoadError("string", 1, UndefVarError(:Plots))
└ @ Main.anonymous none:38



md5-4feb50f51be10f91b1d12465d718843b



```julia
┌ Debug: Error executing precompile(Tuple{Type{Measures.Length{:pct, Float64}}, Float64}):
│ LoadError("string", 1, UndefVarError(:Measures))
└ @ Main.anonymous none:38

I'm not sure why these ones failed when most of the statements include a module like this, and succeed. So there's something weird going on there. But _probably_ this is something internal to PackageCompiler?

[timholy]

For your case 4, these are likely due to partial inference (e.g., from being called by a @nspecialize method, specialization heuristics, or outright inference failure) or module-level @nospecialize. For example, Tuple{typeof(Base.Docs.docstr), Any, Any} comes from this @nospecialize which applies to the entire Docs module.

But I do think Julia should not complain about precompiling that signature.

[NHDaly]

Oh _interesting_, thanks!

But I do think Julia should not complain about precompiling that signature.

Yeah, agreed! this seems like it might be a bug. I've checked, and these statements are emitted _again_ when re-running my snoop file with the resulting sysimg, so I do not believe they are being statically compiled.

Thanks for the context, @timholy

[NHDaly]

Yeah, indeed I think Case 4 is a bug in julia. I've opened an issue for it here:
https://github.com/JuliaLang/julia/issues/38149

[NHDaly]

(okay and indeed Case 5 was related to PackageCompiler; i've opened a PR here: https://github.com/JuliaLang/PackageCompiler.jl/pull/463)

[NHDaly]

Alright, and finally Class 6 -- this is the last of the errors we're currently seeing in our usage, I think, although it's also the biggest:

Class 6: precompile _succeeds_, but somehow isn't serialized during static compilation.

This last class of issue is for statements that appear to precompile() successfully, yet doing so during sysimg construction _does not_ get cached as you would expect. Here is an example:

precompile(Tuple{typeof(Base.push!), Array{Function, 1}, Function})

This function shows up in the --trace-compile output both during snoop compiling and again when using the newly created sysimg:

julia> using PackageCompiler

julia> mktemp() do f, io
           write(io, "precompile(Tuple{typeof(Base.push!), Array{Function, 1}, Function})")
           close(io)
           PackageCompiler.create_sysimage(sysimage_path="./precompiled", precompile_statements_file=f)
       end
[ Info: PackageCompiler: creating system image object file, this might take a while...
julia>

$ julia --trace-compile=stderr -J./precompiled -e 'precompile(Tuple{typeof(Base.push!), Array{Function, 1}, Function})'
precompile(Tuple{typeof(Base.MainInclude.include), String})
precompile(Tuple{typeof(Base.push!), Array{Function, 1}, Function})



md5-8db414987aae6615e926fbeed524c923



as seen here:



md5-2781d446ba1b8a1f4cb4832cb58a54fd



```bash
julia -J./precompiled --trace-compile=stderr -e '
               precompile(Tuple{typeof(Base.:(&)), Int64, Int64})                   # works as expected
               precompile(Tuple{typeof(Base.push!), Array{Function, 1}, Function})  # doesnt cache
'
precompile(Tuple{typeof(Base.MainInclude.include), String})
precompile(Tuple{typeof(Base.push!), Array{Function, 1}, Function})

Out of the 15627 precompile statements emitted, we had * 2035 cases* of this! In total, including all of the above cases, we have 2629 failures/errors, and in practice we find that only about 50-60% of the compilation time is recovered by using a statically compiled binary. I am assuming that this last class of failures is the bulk of the problem, but I don't yet know why.

[KristofferC]

Could Case 6 be from invalidations or type piracy?

Have you tried the "double sysimage" trick (https://github.com/JuliaLang/PackageCompiler.jl/issues/429)? Basically, you compile a sysimage, and then using that sysimage as the base, you compile the sysimage again. At that point, all invalidations should have been resolved.

[NHDaly]

Oh, wow! Thanks for the link; I hadn't seen that. That's very fascinating.

Could it explain a very simple case like the one I show above, where we're literally just precompiling the one single function?

Is the idea that i should then run PackageCompiler _starting from_ the sysimg and compile a new sysimg?

[KristofferC]

Is the idea that i should then run PackageCompiler starting from the sysimg and compile a new sysimg?

Yes, and use incremental=true. Note that this is just a theory but it seems this has helped other people, so worth trying.

[NHDaly]

Hrmm no dice. :(

I ran julia -J./precompiled, then re-did the create_sysimage() from above, but this time I added incremental=true, and still the new process prints the same trace-compile for only precompile(Tuple{typeof(Base.push!), Array{Function, 1}, Function}). :(

Thanks for the info though; i'll look into applying this to our build overall. (though currently static compilation takes about an hour, so running it twice is a tough pill to swallow... 😬)