Julia: static compilation issues

[SimonDanisch]

Since most issues I encounter are Julia bugs, it seems to make sense to have an issue tracking the various problems I'm running into when compiling a reasonably large Julia library with static-julia.

Right now I'm trying to target Julia 0.6.1. Is this my first big mistake, and 0.6 is a lost battle for static compilation?

Some segfaults I get:

Segfault in:

function uniformlocations(nametypedict::Dict{Symbol, GLenum}, program)
    isempty(nametypedict) && return Dict{Symbol,Tuple}()
    texturetarget = -1 # start -1, as texture samplers start at 0
    return Dict{Symbol, Tuple}(map(nametypedict) do name_type
        name, typ = name_type
        loc = get_uniform_location(program, name)
        str_name = string(name)
        if istexturesampler(typ)
            texturetarget += 1
            return name => (loc, texturetarget)
        else
            return name => (loc,)
        end
    end)
end

A lot of references to type inferance, so maybe the Abstract element type is confusing something?
Fixed by just directly adding keys to Dict{Symbol, Tuple}:
GDB stack trace: https://gist.github.com/SimonDanisch/d6445c9e6a225369674e1195523e2228#file-gistfile1-txt

In char = SubString(string, i, i).
I'm not sure why I aws using a Substring to extract a single char in the first place, but when I replaced this with indexing directly into the string, I found out that I had a hidden bounds error. Maybe that got unmasked here! Directly indexing + removing bounds error let me move on.
GDB stack trace: https://gist.github.com/SimonDanisch/9d1fe9414cca52b407e2c535ccf0502e#file-substring-txt

Various segfault point to task.c without any stack trace:

(gdb) run
Starting program: /home/s/juliastuff/static-julia/builddir/hello 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff0642700 (LWP 25631)]
,,
[New Thread 0x7fffaedb1700 (LWP 25645)]
WARNING: 
Thread 1 "hello" received signal SIGSEGV, Segmentation fault.
0x00007ffff3dfbdeb in ctx_switch (ptls=<error reading variable: Cannot access memory at address 0xffffffffffffffc8>, t=<error reading variable: Cannot access memory at address 0xffffffffffffffc0>, 
    where=<error reading variable: Cannot access memory at address 0xffffffffffffffb8>) at /home/s/juliastuff/julia06dev/src/task.c:360
360             asm(" movq %0, %%rsp;\n"
(gdb) backtrace
#0  0x00007ffff3dfbdeb in ctx_switch (ptls=<error reading variable: Cannot access memory at address 0xffffffffffffffc8>, t=<error reading variable: Cannot access memory at address 0xffffffffffffffc0>, 
    where=<error reading variable: Cannot access memory at address 0xffffffffffffffb8>) at /home/s/juliastuff/julia06dev/src/task.c:360
Backtrace stopped: Cannot access memory at address 0x8
(gdb) 

It seems to try to print a warning - this might actually happen in an init function when I think about it. Would that be unsafe?

[SimonDanisch]

Another one:
https://gist.github.com/SimonDanisch/0aab7d494bd1800e161cdc23ac5d40d9
Again, dictionaries are involved: https://github.com/JuliaGL/GLWindow.jl/blob/sd/static/src/events.jl#L28... Not sure if they're problematic because they're the main source of type instability or if dictionaries themselves are the problem.

[JeffBezanson]

What steps are needed to reproduce the segfault?

[SimonDanisch]

Very good question. I tried creating an MWE but have failed so far. I can see if I can put together an example script using all my packages.

Btw, I went back to work on the package and I suddenly also got some assertion errors, but only with julia-debug. So maybe some of the above are not even specific to static compilation:

s@s-d:~/.julia/v0.6/MakiE$ julia-debug docs/make.jl 
Documenter: setting up build directory.
Documenter: expanding markdown templates.
Documenter: building cross-references.
Documenter: running document checks.
 > checking for missing docstrings.
julia: /home/s/juliastuff/julia06dev/src/subtype.c:2284: tuple_morespecific: Assertion `!type_morespecific_(ce, pe, invariant, env)' failed.

signal (6): Aborted
while loading /home/s/.julia/v0.6/MakiE/docs/make.jl, in expression starting on line 3
raise at /build/glibc-bfm8X4/glibc-2.23/signal/../sysdeps/unix/sysv/linux/raise.c:54
abort at /build/glibc-bfm8X4/glibc-2.23/stdlib/abort.c:89
__assert_fail_base at /build/glibc-bfm8X4/glibc-2.23/assert/assert.c:92
__assert_fail at /build/glibc-bfm8X4/glibc-2.23/assert/assert.c:101
tuple_morespecific at /home/s/juliastuff/julia06dev/src/subtype.c:2284
type_morespecific_ at /home/s/juliastuff/julia06dev/src/subtype.c:2438
jl_type_morespecific at /home/s/juliastuff/julia06dev/src/subtype.c:2587
jl_typemap_list_insert_sorted at /home/s/juliastuff/julia06dev/src/typemap.c:1088
jl_typemap_list_insert_ at /home/s/juliastuff/julia06dev/src/typemap.c:924
jl_typemap_level_insert_ at /home/s/juliastuff/julia06dev/src/typemap.c:995
jl_typemap_insert_generic at /home/s/juliastuff/julia06dev/src/typemap.c:933
jl_typemap_array_insert_ at /home/s/juliastuff/julia06dev/src/typemap.c:954
jl_typemap_level_insert_ at /home/s/juliastuff/julia06dev/src/typemap.c:992
jl_typemap_insert_generic at /home/s/juliastuff/julia06dev/src/typemap.c:933
jl_typemap_array_insert_ at /home/s/juliastuff/julia06dev/src/typemap.c:954
jl_typemap_level_insert_ at /home/s/juliastuff/julia06dev/src/typemap.c:992
jl_typemap_insert_generic at /home/s/juliastuff/julia06dev/src/typemap.c:933
jl_typemap_insert at /home/s/juliastuff/julia06dev/src/typemap.c:1060
cache_method at /home/s/juliastuff/julia06dev/src/gf.c:1019
jl_mt_assoc_by_type at /home/s/juliastuff/julia06dev/src/gf.c:1062
jl_lookup_generic_ at /home/s/juliastuff/julia06dev/src/gf.c:1894
jl_apply_generic at /home/s/juliastuff/julia06dev/src/gf.c:1923
push! at ./set.jl:35 [inlined]
union! at ./set.jl:64
allbindings at /home/s/.julia/v0.6/Documenter/src/DocChecks.jl:69
allbindings at /home/s/.julia/v0.6/Documenter/src/DocChecks.jl:58
missingdocs at /home/s/.julia/v0.6/Documenter/src/DocChecks.jl:30
runner at /home/s/.julia/v0.6/Documenter/src/Builder.jl:188
dispatch at /home/s/.julia/v0.6/Documenter/src/Selectors.jl:164
cd at ./file.jl:70
#makedocs#1 at /home/s/.julia/v0.6/Documenter/src/Documenter.jl:186
unknown function (ip: 0x7f19000a38d0)
jl_call_fptr_internal at /home/s/juliastuff/julia06dev/src/julia_internal.h:339
jl_call_method_internal at /home/s/juliastuff/julia06dev/src/julia_internal.h:358
jl_apply_generic at /home/s/juliastuff/julia06dev/src/gf.c:1926
#makedocs at ./<missing>:0
unknown function (ip: 0x7f19000a3582)
jl_call_fptr_internal at /home/s/juliastuff/julia06dev/src/julia_internal.h:339
jl_call_method_internal at /home/s/juliastuff/julia06dev/src/julia_internal.h:358
jl_apply_generic at /home/s/juliastuff/julia06dev/src/gf.c:1926
do_call at /home/s/juliastuff/julia06dev/src/interpreter.c:75
eval at /home/s/juliastuff/julia06dev/src/interpreter.c:242
jl_interpret_toplevel_expr at /home/s/juliastuff/julia06dev/src/interpreter.c:34
jl_toplevel_eval_flex at /home/s/juliastuff/julia06dev/src/toplevel.c:577
jl_parse_eval_all at /home/s/juliastuff/julia06dev/src/ast.c:873
jl_load at /home/s/juliastuff/julia06dev/src/toplevel.c:616
jl_load_ at /home/s/juliastuff/julia06dev/src/toplevel.c:623
include_from_node1 at ./loading.jl:576
unknown function (ip: 0x7f190f919d0b)
jl_call_fptr_internal at /home/s/juliastuff/julia06dev/src/julia_internal.h:339
jl_call_method_internal at /home/s/juliastuff/julia06dev/src/julia_internal.h:358
jl_apply_generic at /home/s/juliastuff/julia06dev/src/gf.c:1926
include at ./sysimg.jl:14
unknown function (ip: 0x7f190f70ce9b)
jl_call_fptr_internal at /home/s/juliastuff/julia06dev/src/julia_internal.h:339
jl_call_method_internal at /home/s/juliastuff/julia06dev/src/julia_internal.h:358
jl_apply_generic at /home/s/juliastuff/julia06dev/src/gf.c:1926
process_options at ./client.jl:305
_start at ./client.jl:371
unknown function (ip: 0x7f190f92cf98)
jl_call_fptr_internal at /home/s/juliastuff/julia06dev/src/julia_internal.h:339
jl_call_method_internal at /home/s/juliastuff/julia06dev/src/julia_internal.h:358
jl_apply_generic at /home/s/juliastuff/julia06dev/src/gf.c:1926
unknown function (ip: 0x401719)
unknown function (ip: 0x401dc0)
unknown function (ip: 0x402293)
__libc_start_main at /build/glibc-bfm8X4/glibc-2.23/csu/../csu/libc-start.c:291
unknown function (ip: 0x401508)
Allocations: 60788154 (Pool: 60778500; Big: 9654); GC: 189
Aborted (core dumped)

[SimonDanisch]

Ahaaa, my first test app actually works if I don't use julia-debug :) :)

But it does actually still take quite a bit of time. I'm guessing that type inference doesn't do something magical like using the function calls and types inside main to precompile more binary ;)

[tknopp]

I think you need precompile statements in order to cache the compilation result. If startup is you concern wouldn't it be easier to use a custom system image?

[SimonDanisch]

Just in this second I got it to work together with SnoopCompile! ~0.5s to get to the above plot :)

I dislike the system image solution, because it's hard to ship to the user - fiddling with the users system image is pretty fragile and if you mess up, the user might not be able to use Julia anymore... It also doesn't compose very well with other packages having the same goal. It really just works for a matlab kind of style of: here are your N packages which will work fast, or for expert users ;) But I'm not really targeting expert users, since they might actually develop these packages, so putting things in the system image isn't helping. I think @timholy Revise is a much better solution in that case.

[tknopp]

to clarify, building a dedicated system image is actually part of static compilation, no?

Whatever you have done, please give step-by-step tutorial somewhere (e.g. in discourse). I have an app needing 4 minutes to startup and still did not get it faster.

[SimonDanisch]

to clarify, building a dedicated system image is actually part of static compilation, no?

Kind of, as I understand it... But it doesn't replace the system image used by julia, it just links it to another executable!

Which is also a bit annoying. So let me figure out some details and discuss our options here - I hope to be able to put my findings into a package when I figured out how to satisfy my use case.

[SimonDanisch]

So my plan is to have something like this:

@static_interface begin
    using MakiE
    scene = Scene()
    scatter(...)
    lines(...)
    volume(...)
end

This will then create a couple of things, e.g. snoop compiling this code block and generating a binary out of it. This would go well with: we only offer precompilation for what we test.

Now the question is, should this then end up in the system image, or should the above also create something like:

module MakiEBinWrap
Scene() = ccall((:Scene, :MakiEBinary), Scene, Tuple{})
scatter(arg) = ccall((:scatter, :MakiEBinary), Scene, Tuple{...}) # types will be found by SnoopCompile
...
end

The above would be really cool, since one could also generate python and C wrapper in a similar manner.

I'm not really sure if it's necessary or if there is an easier option to just get compiled functions back into the julia process, without fiddling with the system image.

@JeffBezanson @Keno, any ideas on this?

[SimonDanisch]

This might also be another use case for BinaryBuilder for me... To ship out tested and dependency independent Julia binaries to the user. So they can just install my package and I/them don't need to worry what version of other Julia packages they have installed. Although I'm not really sure if that's the most elegant solution ;)

[SimonDanisch]

Behold:

The whole stacks seems to work now, all in under 1s :)

[tknopp]

I have been reading a lot about precompulation, snoop compile and so on. Whatever is done here on the prototype level: Please, please integrate this with the package ecosystem (-> Pkg3) so that it gets dead simple to get fast applications. This will solve one of the last big drawbacks of Julia compared to other widespread languages.

[KristofferC]

That's wicked!

[tknopp]

@SimonDanisch: Did you use the current master of SnoopCompile? It has just yesterday been fixed to actually work on 0.6: https://github.com/timholy/SnoopCompile.jl/commits/master

[SimonDanisch]

Yes I was very lucky and that's what I'm using! After switching to master, the snooping experience was very smooth!

[SimonDanisch]

@JeffBezanson I added my script with explicit checkout instructions for GLVisualize: https://github.com/SimonDanisch/static-julia/blob/simon/sd/glvisualize/execute.jl
Without checking out the branches, I run into all the errors above.

[SimonDanisch]

@tknopp, I just finished a first version of PackageCompiler, which combines SnoopCompile and the system image building script to automatically create a system image from the runtest file of a package.
I tested it on windows and linux so far, and might extend it for a few more use cases - Although I'm waiting on finding out more about the direction Julia base is going ;)

[ViralBShah]

@SimonDanisch Should we leave this open? Maybe revisit it once the alpha is tagged?

[SimonDanisch]

I don't see why we should close thise ;) Are there actually any expected improvements?