Pkg.jl: Warning when conditionally loading "glue" module

[DilumAluthge]

Cross-posted from https://github.com/MikeInnes/Requires.jl/issues/65

I can conditionally load code like this:

ExampleOne.jl:

module ExampleOne

using Requires

function __init__()
    @require JSON="682c06a0-de6a-54ab-a142-c8b1cf79cde6" begin
        function hello()
            println("Hello world!")
        end
    end
end

end

But code loaded like this is not precompiled.

If I want to be able to precompile my conditionally loaded code, I can do so like this:

ExampleTwo.jl:

module ExampleTwo

using Requires

function __init__()
    pushfirst!(Base.LOAD_PATH, @__DIR__)
    @require JSON="682c06a0-de6a-54ab-a142-c8b1cf79cde6" using GlueModule
end

end

Where the contents of GlueModule.jl are:

module GlueModule

function hello()
    println("Hello world!")
end

end

With this approach, GlueModule is precompiled. Unfortunately, I get this warning:

julia> using ExampleTwo ┌ Warning: Package ExampleTwo does not have GlueModule in its dependencies: │ - If you have ExampleTwo checked out for development and have │ added GlueModule as a dependency but haven't updated your primary │ environment's manifest file, try `Pkg.resolve()`. │ - Otherwise you may need to report an issue with ExampleTwo └ Loading GlueModule into ExampleTwo from project dependency, future warnings for ExampleTwo are suppressed.

I understand the purpose of the "Warning: Package Foo does not have Bar in its dependencies" warning when Bar is a real package with a UUID that is in the Julia General registry and can be added to a Project.toml file. But in this case, GlueModule isn't a real package - it's a glue module that is located in the same package repository as ExampleTwo. GlueModule doesn't have its own GitHub repository, doesn't have its own UUID, is not separately registered in the Julia General registry, and cannot be added to a Project.toml file.

So is there a way to suppress the "Warning: Package ExampleTwo does not have GlueModule in its dependencies" warning for glue modules that are conditionally loaded by Requires.jl?

[KristofferC]

Why not include GlueModule.jl and do using .GlueModule?

Anyway, the package manager is not involved in this at all. The way code is loaded is implemented in Julia itself.

[MikeInnes]

Should this become a Julia issue then? It may be that the only way to resolve this is to change how precompilation works somehow, but I figured there's some chance that we can just make Pkg aware of a sub-package, which Julia already knows how to precompile.

[KristofferC]

I don't know what a "sub-packages" is but still, no code from Pkg.jl is executed when loading a package. Anyway, touching the global LOAD_PATH in __init__ seems very dubious.

Again, what happens on

@require JSON="682c06a0-de6a-54ab-a142-c8b1cf79cde6" begin
    include("GlueModule.jl")
    using .GlueModule
end

?

[MikeInnes]

That's fine except that it isn't precompiled. So the workaround that @DilumAluthge is proposing is to have a folder structure like

* MyPackage/
  * GlueModule/
    * src/GlueModule.jl
    * Project.toml
  * src/MyPackage.jl
  * Project.toml

If GlueModule is on the load path you can load it with precompilation. This all works well with the only issue that a warning comes up because GlueModule doesn't appear in MyPackage's Project.toml. So the question here is really whether we can avoid that warning, perhaps by listing GlueModule in extras or something similar (though this doesn't appear to work).

Does that make things clearer?

[KristofferC]

perhaps by listing GlueModule in extras or something similar (though this doesn't appear to work).

The only section that code loading (defined in Julia itself) looks at when loading a dependency inside a package is the [deps] entry in the Manifest.

Pushing things to LOAD_PATH and conditionally loading packages in there inside __init__ is not really how anything has been designed to work so it is not surprising that things are not going so smoothly.

[MikeInnes]

It's kind of unclear whether you're saying that this should be an issue on the Julia repo, as I asked before, or just a wontfix. I guess the latter.

[DilumAluthge]

Pushing things to LOAD_PATH and conditionally loading packages in there inside __init__ is not really how anything has been designed to work so it is not surprising that things are not going so smoothly.

Hmmm. Then what is the recommended way of conditionally loading code that takes advantage of precompilation?

[DilumAluthge]

It seems like this isn’t a Pkg issue, so I have cross posted here: https://github.com/JuliaLang/julia/issues/32413