Flux.jl: Crashes once compiled to sysimage with PackageCompiler.jl

[xiaodaigh]

Firstly compile both Flux.jl and CUDA.jl

using PackageCompiler
compile_sysimage([:Flux, :CUDA], replace_default=true)

Now restart Julia and run this

using Flux, CUDA

data = rand(28, 28) |> gpu

model = Chain(
    x->reshape(x, 28, 28, 1, :),
    Conv((5,5), 1=>8; stride = 2, pad = 1),
) |> gpu

model(data)

It fails with the below error which does not happen if I do not compile CUDA.jl and Flux.jl. In fact, compiling just Flux.jl alone doesn't cause this error.

I didn't know where to file this but @ViralBShah had asked me to file it here.

Flux v0.11.1 CUDA v1.3.3

julia> versioninfo()
Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = "C:\Users\RTX2080\AppData\Local\Programs\Microsoft VS Code\Code.exe"
  JULIA_NUM_THREADS = 6
  JULIA_PKG_DEVDIR = c:/git/

crash printout

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6689f445 -- has_free_typevars at /cygdrive/d/buildbot/worker/package_win64/build/src\jltypes.c:166
in expression starting at REPL[3]:1
has_free_typevars at /cygdrive/d/buildbot/worker/package_win64/build/src\jltypes.c:166
jl_has_free_typevars at /cygdrive/d/buildbot/worker/package_win64/build/src\jltypes.c:186 [inlined]
inst_type_w_ at /cygdrive/d/buildbot/worker/package_win64/build/src\jltypes.c:1558
jl_instantiate_unionall at /cygdrive/d/buildbot/worker/package_win64/build/src\jltypes.c:918 [inlined]
jl_wrap_Type at /cygdrive/d/buildbot/worker/package_win64/build/src\jltypes.c:1607
arg_type_tuple at /cygdrive/d/buildbot/worker/package_win64/build/src\gf.c:1842
jl_tuple1_isa at /cygdrive/d/buildbot/worker/package_win64/build/src\subtype.c:2021
jl_typemap_entry_assoc_exact at /cygdrive/d/buildbot/worker/package_win64/build/src\typemap.c:679
jl_typemap_level_assoc_exact at /cygdrive/d/buildbot/worker/package_win64/build/src\typemap.c:720
jl_typemap_assoc_exact at /cygdrive/d/buildbot/worker/package_win64/build/src\julia_internal.h:1079 [inlined]
jl_lookup_generic_ at /cygdrive/d/buildbot/worker/package_win64/build/src\gf.c:2342
jl_apply_generic at /cygdrive/d/buildbot/worker/package_win64/build/src\gf.c:2394
jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined]
_hook_call<1> at /cygdrive/d/buildbot/worker/package_win64/build/src\cgutils.cpp:134
jl_setup_module at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:1078 [inlined]
emit_function at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:5138
jl_emit_code at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:6431
jl_create_native at /cygdrive/d/buildbot/worker/package_win64/build/src\aotcompile.cpp:317
compile_method_instance at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\irgen.jl:123
macro expansion at C:\Users\RTX2080\.julia\packages\TimerOutputs\dVnaw\src\TimerOutput.jl:206 [inlined]
irgen at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\irgen.jl:332
macro expansion at C:\Users\RTX2080\.julia\packages\TimerOutputs\dVnaw\src\TimerOutput.jl:206 [inlined]
macro expansion at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\driver.jl:101 [inlined]
macro expansion at C:\Users\RTX2080\.julia\packages\TimerOutputs\dVnaw\src\TimerOutput.jl:206 [inlined]
#codegen#80 at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\driver.jl:100
codegen##kw at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\driver.jl:63
unknown function (ip: 0000000022E8CEC1)
emit_function! at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\rtlib.jl:77
unknown function (ip: 0000000022E8C8C5)
build_runtime at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\rtlib.jl:117
#65 at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\rtlib.jl:159
get! at .\dict.jl:450
load_runtime at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\rtlib.jl:151
#codegen#80 at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\driver.jl:96
codegen##kw at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\driver.jl:63 [inlined]
#compile#78 at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\driver.jl:39
compile at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\driver.jl:35 [inlined]
#_cufunction#779 at C:\Users\RTX2080\.julia\packages\CUDA\dZvbp\src\compiler\execution.jl:310
_cufunction at C:\Users\RTX2080\.julia\packages\CUDA\dZvbp\src\compiler\execution.jl:304 [inlined]
#check_cache#99 at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\cache.jl:24
unknown function (ip: 0000000022E7D4F8)
check_cache##kw at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\cache.jl:11
unknown function (ip: 0000000022E7C32C)
broadcast_kernel at C:\Users\RTX2080\.julia\packages\GPUArrays\f3yak\src\host\broadcast.jl:60 [inlined]
#cached_compilation#100 at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\cache.jl:0
cached_compilation at C:\Users\RTX2080\.julia\packages\GPUCompiler\GKp4B\src\cache.jl:40 [inlined]
#cufunction#778 at C:\Users\RTX2080\.julia\packages\CUDA\dZvbp\src\compiler\execution.jl:298
cufunction at C:\Users\RTX2080\.julia\packages\CUDA\dZvbp\src\compiler\execution.jl:293 [inlined]
#launch_heuristic#838 at C:\Users\RTX2080\.julia\packages\CUDA\dZvbp\src\gpuarrays.jl:19 [inlined]
launch_heuristic at C:\Users\RTX2080\.julia\packages\CUDA\dZvbp\src\gpuarrays.jl:17 [inlined]
copyto! at C:\Users\RTX2080\.julia\packages\GPUArrays\f3yak\src\host\broadcast.jl:66 [inlined]
copyto! at .\broadcast.jl:886 [inlined]
copy at .\broadcast.jl:862 [inlined]
materialize at .\broadcast.jl:837
unknown function (ip: 0000000022E7AC50)
Conv at C:\Users\RTX2080\.julia\packages\Flux\05b38\src\layers\conv.jl:147
applychain at C:\Users\RTX2080\.julia\packages\Flux\05b38\src\layers\basic.jl:36
applychain at C:\Users\RTX2080\.julia\packages\Flux\05b38\src\layers\basic.jl:36 [inlined]
Chain at C:\Users\RTX2080\.julia\packages\Flux\05b38\src\layers\basic.jl:38
unknown function (ip: 0000000022E65A86)
jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined]
do_call at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:117
eval_value at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:206
eval_stmt_value at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:157 [inlined]
eval_body at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:548
jl_interpret_toplevel_thunk at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:660
jl_toplevel_eval_flex at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:840
jl_toplevel_eval_flex at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:790
jl_toplevel_eval_flex at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:790
jl_toplevel_eval_flex at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:790
jl_toplevel_eval at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:849 [inlined]
jl_toplevel_eval_in at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:883
eval at .\boot.jl:331
eval_user_input at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\REPL\src\REPL.jl:134
repl_backend_loop at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\REPL\src\REPL.jl:195
start_repl_backend at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\REPL\src\REPL.jl:180
#run_repl#37 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\REPL\src\REPL.jl:292
run_repl at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\REPL\src\REPL.jl:288
#806 at .\client.jl:399
jfptr_YY.806_54633 at C:\Users\RTX2080\AppData\Local\Programs\Julia 1.5.1\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined]
do_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\builtins.c:655
jl_f__apply at /cygdrive/d/buildbot/worker/package_win64/build/src\builtins.c:669 [inlined]
jl_f__apply_latest at /cygdrive/d/buildbot/worker/package_win64/build/src\builtins.c:705
#invokelatest#1 at .\essentials.jl:710 [inlined]
invokelatest at .\essentials.jl:709 [inlined]
run_main_repl at .\client.jl:383
exec_options at .\client.jl:313
_start at .\client.jl:506
jfptr__start_38524 at C:\Users\RTX2080\AppData\Local\Programs\Julia 1.5.1\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined]
true_main at /cygdrive/d/buildbot/worker/package_win64/build/ui\repl.c:106
wmain at /cygdrive/d/buildbot/worker/package_win64/build/ui\repl.c:227
__tmainCRTStartup at /usr/src/debug/mingw64-x86_64-runtime-7.0.0-1/crt\crtexe.c:334
mainCRTStartup at /usr/src/debug/mingw64-x86_64-runtime-7.0.0-1/crt\crtexe.c:223
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
Allocations: 31858795 (Pool: 31851435; Big: 7360); GC: 37

[ViralBShah]

@DhairyaLGandhi Once reproduce this, we should work with @maleadt and @KristofferC to figure out if there is an issue package compiling Flux + GPU and get it fixed.

Ah it's windows - so doubly exciting.

[maleadt]

Anecdotally, I did happen to test CUDA.jl#master with PackageCompiler on Julia 1.5 today and that worked fine (except for it downloading all artifacts), albeit in app mode and with an empty precompile.jl.

That said, llvmcall isn't precompilable on Julia 1.5, so if any GPU kernels got embedded in the system image that will have caused the crash. That should be fixed on 1.6, and may be the underlying issue here. Note that CUDA.jl isn't easily usable on 1.6 right now (needs, at least, https://github.com/JuliaGPU/GPUCompiler.jl/pull/69).

[cjw199]

@maleadt Hi. Assuming that is the source of the problem, would this be resolved only with the latest CUDA.jl on Julia 1.6? Since Flux v0.11.1 requires CUDA v1, I assume that even with Julia 1.6, any code using GPUs with Flux will still be un-precompilable?

[maleadt]

CUDA doesn't yet support Julia 1.6, so that combination won't work anyway. But yes, I expect this to work with an upcoming version of CUDA.jl and Julia 1.6, although there may be additional changes required.

[DhairyaLGandhi]

We would bump the dependency with the existing PR #1345 as well