Julia: `using StatPlots` precompiles package despite `precompile(false)`

Created on 2 Oct 2018 · 11Comments · Source: JuliaLang/julia

Moving this issue over to julia, since it was possibly incorrectly reported as a Pkg bug. Original bug here:
https://github.com/JuliaLang/Pkg.jl/issues/722

StatPlots precompiles every time it is loaded:
julia> using StatPlots
[ Info: Precompiling StatPlots [60ddc479-9b66-56df-82fc-76a74619b69c]
[ Info: Recompiling stale cache file /home/pietro/.julia/compiled/v1.0/Plots/ld3vC.ji for Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
But precompilation of the module is explicitly disabled, see here.

It seems that this should be reported in julia, not Pkg:

Since I am not seeing a single Pkg command there, I assume this is a Julia bug?

packages

Source

NHDaly

Most helpful comment

Every time!

KristofferC on 18 Sep 2019

😄2 ❤1

All 11 comments

Ah, okay, I've done a bit more digging and I now understand the behavior. Everything is working correctly here, it seems, it's just that the _Info logs_ are misleading!

Here is the source of our concern:

It seems that even though StatPlots is registered with __precompile__(false), it still precompiles _every time_ you import it:

julia1.0> using StatPlots
[ Info: Precompiling StatPlots [60ddc479-9b66-56df-82fc-76a74619b69c]

However, I've checked manually, and Base.compilecache() is _indeed_ exiting out immediately, since this package has opted out of precompilation. But it's strange, because it seems that when called through using, it's still precompiling:

julia1.0> @time Base.compilecache(Base.PkgId(Base.UUID("60ddc479-9b66-56df-82fc-76a74619b69c"), "StatPlots"))
[ Info: Precompiling StatPlots [60ddc479-9b66-56df-82fc-76a74619b69c]
  1.034803 seconds (1.50 M allocations: 73.634 MiB, 1.26% gc time)
>> Declaring __precompile__(false) is not allowed in files that are being precompiled.

julia1.0> @time using StatPlots
[ Info: Precompiling StatPlots [60ddc479-9b66-56df-82fc-76a74619b69c]
 17.127503 seconds (38.25 M allocations: 1.951 GiB, 5.38% gc time)

Those 17 seconds _every time_ are frustrating!

However, in fact, everything _is_ working correctly, in that when it attempts to precompile, it sees that it cannot, so it falls back to "just load the file normally":
https://github.com/JuliaLang/julia/blob/d769ad23ee477fc9371f15c1884637da07543c10/base/loading.jl#L977

I separated that logic out into a function, and verified that almost all of those 17seconds are just spent in that "normal" loading:

julia> function normal_load(pkg, path)
               __toplevel__ = Base.__toplevel__
               # just load the file normally via include
               # for unknown dependencies
               uuid = pkg.uuid
               uuid = (uuid === nothing ? (UInt64(0), UInt64(0)) : convert(NTuple{2, UInt64}, uuid))
               old_uuid = ccall(:jl_module_uuid, NTuple{2, UInt64}, (Any,), __toplevel__)
               if uuid !== old_uuid
                   ccall(:jl_set_module_uuid, Cvoid, (Any, NTuple{2, UInt64}), __toplevel__, uuid)
               end
               try
                   Base.include_relative(__toplevel__, path)
                   return
               finally
                   if uuid !== old_uuid
                       ccall(:jl_set_module_uuid, Cvoid, (Any, NTuple{2, UInt64}), __toplevel__, old_uuid)
                   end
               end
       end
normal_load (generic function with 1 method)

julia> pkg = Base.PkgId(Base.UUID("60ddc479-9b66-56df-82fc-76a74619b69c"), "StatPlots")
StatPlots [60ddc479-9b66-56df-82fc-76a74619b69c]

julia> path = Base.locate_package(pkg)
"/Users/daly/.julia/packages/StatPlots/acOno/src/StatPlots.jl"

julia> @time normal_load(pkg, path)
 16.529569 seconds (37.14 M allocations: 1.898 GiB, 5.07% gc time)

So actually, I think the problem is just that maybe we should be more proactive about what kind of Info log we print. Perhaps we can proactively check whether this Module has disabled precompiling before printing that Precompiling $pkg message? Maybe it could say something like Importing $pkg in that case instead or something? Or even just nothing?

NHDaly on 2 Oct 2018

👍2

Any current fix for the issue?

Nosferican on 14 Nov 2018

👍1

We think it might be fixed in StatPlots, but we can't tag a release because CIBot tells us that a depending package has used StatPlots in a nonstandard way and now it breaks...

mkborregaard on 15 Nov 2018

but we can't tag a release because CIBot tells us that a depending package has used StatPlots in a nonstandard way and now it breaks...

This just mean that you cannot automatically tag a new release, no?

KristofferC on 15 Nov 2018

Yeah I hope we can make it go through

mkborregaard on 16 Nov 2018

Thanks for the reminder, I've opened #30064 so that at least there's clearer log message output for this case!

NHDaly on 17 Nov 2018

I've also encountered this problem (related or same bug #28384 ) .
Despite using __precompile__(false) packages will still precompile.

using Mypackage

will always precompile, whereas

include("mypackage.jl")
using .Mypackage

will not, which is probably intended for local packages. This is particularly annoying since the package I'm developing is not (yet) precompilable, so each time I'm using it during development I get those precompilation errors.