Flux.jl precompile takes 20-40 minutes

[LaurCon]

When running

using Flux

the precompiling takes 20-40 minutes on my machine. It never crashes, it just hangs on

[ Info: Precompiling Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]

until the precompiling finishes.

julia> versioninfo() Julia Version 1.4.2 Commit 44fa15b150* (2020-05-23 18:35 UTC) Platform Info: OS: Windows (x86_64-w64-mingw32) CPU: Intel(R) Core(TM) i3-4030U CPU @ 1.90GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-8.0.1 (ORCJIT, haswell) Environment: JULIA_NUM_THREADS = 2

(@v1.4) pkg> st Status C:\Users\Lauren\.julia\environments\v1.4\Project.toml [c52e3926] Atom v0.12.14 [717857b8] DSP v0.6.7 [0c46a032] DifferentialEquations v6.14.0 [31c24e10] Distributions v0.23.4 [587475ba] Flux v0.10.4 [f6369f11] ForwardDiff v0.10.10 [f67ccb44] HDF5 v0.13.2 [7073ff75] IJulia v1.21.1 [f4508453] InfoZIP v0.2.0 [e5e0dc1b] Juno v0.8.2 [23992714] MAT v0.8.0 [f91b31a4] MagNav v0.1.0 #master (..\..\..\Documents\AIIA_NDE\MagNav.jl) [91a5bcdd] Plots v1.2.4 [d330b81b] PyPlot v2.9.0 [a5390f91] ZipFile v0.9.2 [37e2e46d] LinearAlgebra
@ChrisRackauckas

[ViralBShah]

That's insane.

[ChrisRackauckas]

It looks like a slightly older version of Flux. Can we see what happens with the recent release?

]activate TestEnvironment
]add [email protected]
using Flux

I don't think it will fix it, but it's worth a check.

[ViralBShah]

I'm trying 0.11, and it took 184 seconds for me on Julia 1.5-rc1 to precompile. After the precompile, subsequent using Flux statements take 23 sec. This is on core i5.

[ChrisRackauckas]

I wonder if it's as simple as, LLVM just hasn't optimized at all for Core i3 so it hits a weird super slow case. We might need to bring someone else in on this since it seems very specific.

[LaurCon]

After 3 minutes, this is what I get:

julia> using Flux
[ Info: Precompiling Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]
ERROR: LoadError: LoadError: could not load symbol "LLVMGetHostCPUFeatures":
The specified procedure could not be found. 
Stacktrace:
 [1] top-level scope at C:\Users\LaurenConger\.julia\packages\VectorizationBase\xuYhK\src\cpu_info_x86_llvm.jl:4
 [2] include(::Module, ::String) at .\Base.jl:377
 [3] include(::String) at C:\Users\LaurenConger\.julia\packages\VectorizationBase\xuYhK\src\VectorizationBase.jl:1
 [4] top-level scope at C:\Users\LaurenConger\.julia\packages\VectorizationBase\xuYhK\src\VectorizationBase.jl:245
 [5] include(::Module, ::String) at .\Base.jl:377
 [6] top-level scope at none:2
 [7] eval at .\boot.jl:331 [inlined]
 [8] eval(::Expr) at .\client.jl:449
 [9] top-level scope at .\none:3
in expression starting at C:\Users\LaurenConger\.julia\packages\VectorizationBase\xuYhK\src\cpu_info_x86_llvm.jl:4
in expression starting at C:\Users\LaurenConger\.julia\packages\VectorizationBase\xuYhK\src\VectorizationBase.jl:243
ERROR: LoadError: Failed to precompile VectorizationBase [3d5dd08c-fd9d-11e8-17fa-ed2836048c2f] to C:\Users\LaurenConger\.julia\compiled\v1.4\VectorizationBase\Dto5m_Dhpe9.ji.
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1272
 [3] _require(::Base.PkgId) at .\loading.jl:1029
 [4] require(::Base.PkgId) at .\loading.jl:927
 [5] require(::Module, ::Symbol) at .\loading.jl:922
 [6] include(::Module, ::String) at .\Base.jl:377
 [7] top-level scope at none:2
 [8] eval at .\boot.jl:331 [inlined]
 [9] eval(::Expr) at .\client.jl:449
 [10] top-level scope at .\none:3
in expression starting at C:\Users\LaurenConger\.julia\packages\LoopVectorization\9Ft4H\src\LoopVectorization.jl:7
ERROR: LoadError: Failed to precompile LoopVectorization [bdcacae8-1622-11e9-2a5c-532679323890] to C:\Users\LaurenConger\.julia\compiled\v1.4\LoopVectorization\4TogI_Dhpe9.ji.
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1272
 [3] _require(::Base.PkgId) at .\loading.jl:1029
 [4] require(::Base.PkgId) at .\loading.jl:927
 [5] require(::Module, ::Symbol) at .\loading.jl:922
 [6] include(::Module, ::String) at .\Base.jl:377
 [7] top-level scope at none:2
 [8] eval at .\boot.jl:331 [inlined]
 [9] eval(::Expr) at .\client.jl:449
 [10] top-level scope at .\none:3
in expression starting at C:\Users\LaurenConger\.julia\packages\Zygote\iFibI\src\Zygote.jl:14
ERROR: LoadError: Failed to precompile Zygote [e88e6eb3-aa80-5325-afca-941959d7151f] to C:\Users\LaurenConger\.julia\compiled\v1.4\Zygote\4kbLI_Dhpe9.ji.
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1272
 [3] _require(::Base.PkgId) at .\loading.jl:1029
 [4] require(::Base.PkgId) at .\loading.jl:927
 [5] require(::Module, ::Symbol) at .\loading.jl:922
 [6] include(::Module, ::String) at .\Base.jl:377
 [7] top-level scope at none:2
 [8] eval at .\boot.jl:331 [inlined]
 [9] eval(::Expr) at .\client.jl:449
 [10] top-level scope at .\none:3
in expression starting at C:\Users\LaurenConger\.julia\packages\Flux\IjMZL\src\Flux.jl:7
ERROR: Failed to precompile Flux [587475ba-b771-5e3f-ad9e-33799f191a9c] to C:\Users\LaurenConger\.julia\compiled\v1.4\Flux\QdkVy_Dhpe9.ji.
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1272
 [3] _require(::Base.PkgId) at .\loading.jl:1029
 [4] require(::Base.PkgId) at .\loading.jl:927
 [5] require(::Module, ::Symbol) at .\loading.jl:922

[ChrisRackauckas]

@chriselrod looks like it is a CPU thing.

[ViralBShah]

@LaurCon What is your versioninfo()?

[ChrisRackauckas]

julia> versioninfo() Julia Version 1.4.2 Commit 44fa15b150* (2020-05-23 18:35 UTC) Platform Info: OS: Windows (x86_64-w64-mingw32) CPU: Intel(R) Core(TM) i3-4030U CPU @ 1.90GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-8.0.1 (ORCJIT, haswell) Environment: JULIA_NUM_THREADS = 2

[KristofferC]

https://github.com/JuliaLang/julia/issues/36483 might be related. Also cpu model sensitive compilation time.

[chriselrod]

Okay, it would be great to have a reliable way of checking CPU features. There's an issue here: https://github.com/JuliaLang/julia/issues/36367

Hopefully this works. Had to reproduce @runtime_ccall from LLVM.jl, but can't actually depend on LLVM.jl either because for some reason that library causes errors for some people, e.g. https://github.com/SciML/DifferentialEquations.jl/issues/635
It also causes problems for people who installed Julia in a weird way, which I'm not really working around, except by falling back to CpuId.jl.

So at least one hole still exists:
If someone on an Ice Lake Mac and Julia <= 1.5 installed Julia in a way that they have Julia linked with x LLVM shared libraries where x != 1, it'll fall back to using CpuId.jl, which will report that the CPU has AVX512, ultimately causing a crash because LLVM will disagree.

EDIT:
That said, with the help of @YingboMa (who also helped identify a memory leak), that problem is solved and (fingers crossed) things will hopefully be more robust on x86.

That said, I'll eventually need to find out how to abstract these features in a way that is also compatible with ARM, and then reliably query those as well. Right now, it doesn't do any feature detection if Sys.ARCH isn't :x86_64 or :i686, and just uses some generic (slow) default settings.

[bpesquet]

I apologize for hijacking this issue instead of creating a new, specific one.

I'm trying 0.11, and it took 184 seconds for me on Julia 1.5-rc1 to precompile. After the precompile, subsequent using Flux statements take 23 sec. This is on core i5.

using Flux takes approx. 25s on my 2014 MacBook Pro. As a new user of Julia and Flux, I'm very surprised and, to be frank, quite disappointed by these numbers. Everything else still works fine on this machine.

Is this expected?
Are there any workarounds to shorten the code-compile-test cycle with Flux?

I'm using Flux v0.11.1. Here's my system info:

julia> versioninfo()
Julia Version 1.5.2
Commit 539f3ce943 (2020-09-23 23:17 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i5-4308U CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, haswell)

Thanks in advance for your help.

[ChrisRackauckas]

Generally you can just use Revise.jl. That doesn't mean we shouldn't solve this, but it shouldn't effect your code-compile-test cycle.

[chriselrod]

Are there any workarounds to shorten the code-compile-test cycle with Flux?

The most popular workaround is to use Revise.jl to avoid restarting your Julia session.
It will automatically update packages you've using-ed and scripts you've includet-ed (instead of include) as you modify them.

This time should get better overtime. For example, "time to first gradient" of the new AD Keno has been working on is much better than Zygote's, which Flux currently uses.

[bpesquet]

@ChrisRackauckas @chriselrod Thanks for these super fast answers.

Working in notebooks seems to alleviate the problem. i'll stick to them for now, and will also try Revise.jl.

[ToucheSir]

@bpesquet are you aware of https://github.com/FluxML/Flux.jl/issues/1155? I have no clue if load times are improved on master (if indeed Flux even loads there), but it might be worth a try. Also, a good chunk of Flux's load time (>2520% IIRC) consists of CUDA.jl, so the new AD (Diffractor) is only part of the equation here.

[DhairyaLGandhi]

Master would work, but load times aren't substantially different there. We should do an invalidations pass on master for the load times.

[BaoBaoDeMeng]

julia> using Flux
[ Info: Precompiling Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]
julia>  versioninfo()
Julia Version 1.4.2
Commit 44fa15b150* (2020-05-23 18:35 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
Environment:
  JULIA_PKG_SERVER = https://github.com/FluxML/Flux.jl.git

Can you use it directly afterwards??

[DhairyaLGandhi]

Yes

[BaoBaoDeMeng]

julia> using Flux
julia> using DiffEqFlux
julia> using ModelingToolkit
julia> using DiffEqBase
julia> using Plots
julia> using NeuralNetDiffEq
julia> using Test
@parameters t θ
@variables u(..)
@derivatives Dt'~t
eq = Dt(u(t,θ)) ~ t^3 + 2*t + (t^2)*((1+3*(t^2))/(1+t+(t^3))) - u(t,θ)*(t + ((1+3*(t^2))/(1+t+t^3)))
bcs = [u(0.) ~ 1.0 , u(1.) ~ 1.202]
domains = [t ∈ IntervalDomain(0.0,1.0)]
dx = 0.1
discretization = PhysicsInformedNN(dx)

ERROR: UndefVarError: PhysicsInformedNN not defined
Stacktrace:
How to solve this problem??

[ChrisRackauckas]

That's a different library. You're looking for NeuralPDE.jl:

https://neuralpde.sciml.ai/dev/examples/pinns_example/

[ToucheSir]

这个问题和Flux没有任何关系...我强烈建议你去Discourse问。 问之前读一下https://discourse.julialang.org/t/psa-make-it-easier-to-help-you/14757。

(I think these posts are all safe to be marked off topic :))

[BaoBaoDeMeng]

这个问题和Flux没有任何关系...我强烈建议你去Discourse问。 问之前读一下https://discourse.julialang.org/t/psa-make-it-easier-to-help-you/14757。

(I think these posts are all safe to be marked off topic :))

Thanks I solve it.

[BaoBaoDeMeng]

I meet another question.
`using Flux
using DiffEqFlux
using ModelingToolkit
using DiffEqBase
using Plots
using NeuralPDE
using Test

Example 1, 1D ode

@parameters t θ
@variables u(..)
@derivatives Dt'~t

1D ODE

eq = Dt(u(t,θ)) ~ t^3 + 2t + (t^2)((1+3(t^2))/(1+t+(t^3))) - u(t,θ)(t + ((1+3*(t^2))/(1+t+t^3)))

Boundary conditions

bcs = [u(0.) ~ 1.0 , u(1.) ~ 1.202]

Space and time domains

domains = [t ∈ IntervalDomain(0.0,1.0)]

Discretization

dx = 0.1
discretization = PhysicsInformedNN(dx)`
ERROR: MethodError: no method matching PhysicsInformedNN(::Float64)
Closest candidates are:
PhysicsInformedNN(::Any, ::Any) at C:\Users\YanWei.julia\packages\NeuralPDE\oWaNg\src\pinns_pde_solve.jl:33
PhysicsInformedNN(::Any, ::Any, ::Any; _phi, autodiff, _derivative, strategy, kwargs...) at C:\Users\YanWei.julia\packages\NeuralPDE\oWaNg\src\pinns_pde_solve.jl:33
PhysicsInformedNN(::D, ::C, ::P, ::PH, ::Bool, ::DER, ::T, ::K) where {D, C, P, PH, DER, T, K} at C:\Users\YanWei.julia\packages\NeuralPDE\oWaNg\src\pinns_pde_solve.jl:15
Stacktrace:
How can I solve this problem???

[ToucheSir]

Again, this is a question you should be posting on Discourse and not here. The only reason you should be posting here is if you have the problem "Flux.jl precompile takes 20-40 minutes". If you post an unrelated question, it will either a) be ignored, or b) be deleted. Either way, you are not going to get an answer here, so please ask on Discourse.