Pkg.jl: Feature request: scan subdirs for updated versions/projects

[timholy]

Summary

For repositories that have more than one package on them, and some of the packages depend on others, CI runs into trouble because julia --project picks the "umbrella" Project.toml files which sets a new version just for that one package. If the umbrella package depends on new versions of the subdirectory packages, the resolver doesn't know that it can get them from the current PR.

Current workaround

If you're running CI with GitHub Actions, add something like

defaults:
  run:
    shell: bash
...

    steps:
      ...
      - run: julia --project -e 'using Pkg; Pkg.develop([PackageSpec(path="SubDirA"),
                                                         PackageSpec(path="SubDirB"), ...])'
 ```

Travis users should add something similar.

### Reproducer

Following on the lengthy discussion in #1251, I thought since all the people who actually know Pkg internals were confused about what I was asking, I would create something that is close to an actual test case (and therefore may assist in developing the requested feature). The executive summary here is that Pkg does not yet support updating its awareness of sub-package versions based on a new pull request.

Setup: create a blank directory to do this test in, I chose `/tmp/pkgs`. Save the script below as `/tmp/pkgs/generate_test.jl`, make sure you've `cd /tmp/pkgs`, launch Julia as `JULIA_DEPOT_PATH=/tmp/pkgs julia --startup-file=no` (I was using Julia 1.6 just to get the latest) and then `include` the script. Here's the script:

```julia
function writelines(path, lines)
    open(path, "w") do io
        for line in lines
            println(io, line)
        end
    end
end

# Create a package BigPkg with a subdirectory SubPkg
# Make BigPkg depend on SubPkg
# Register it
using Pkg
Pkg.add("LocalRegistry")
using LocalRegistry
regpath = joinpath(@__DIR__, "registries", "MyRegistry")
create_registry("MyRegistry", regpath)
# Pkg.Registry.add(regpath)
devpath = joinpath(@__DIR__, "dev")
mkpath(devpath)
bigpkgpath = joinpath(devpath, "BigPkg")
subpkgpath = joinpath(bigpkgpath, "SubPkg")
Pkg.generate(bigpkgpath)
Pkg.generate(subpkgpath)
Pkg.develop(path=bigpkgpath)
Pkg.develop(path=subpkgpath)
cd(bigpkgpath) do
    run(`git init`)
    run(`git add Project.toml src/BigPkg.jl SubPkg/Project.toml SubPkg/src/SubPkg.jl`)
    run(`git commit -m "Skeleton"`)

    Pkg.activate(".")
    Pkg.add(path=".", subdir="SubPkg")
    lines = readlines("Project.toml")
    append!(lines, ["", "[compat]", "SubPkg = \"0.1\""])
    writelines("Project.toml", lines)
    lines = readlines("src/BigPkg.jl")
    insert!(lines, 2, "using SubPkg")
    writelines("src/BigPkg.jl", lines)
    Pkg.activate()
    Pkg.resolve()
    run(`git add Project.toml src/BigPkg.jl`)
    run(`git commit -m "Add SubPkg dependency"`)
end
using BigPkg
using SubPkg
register(SubPkg, "MyRegistry"; repo=bigpkgpath)
register(BigPkg, "MyRegistry"; repo=bigpkgpath)

# OK, now both packages are registered at v0.1.
# Let's add a new feature to SubPkg and bump its version
# to 0.2. Simultaneously, let's require the new feature
# in BigPkg and bump its version to 0.2.
cd(subpkgpath) do
    lines = readlines("src/SubPkg.jl")
    insert!(lines, 2, "export f")
    insert!(lines, 3, "f() = 1")
    writelines("src/SubPkg.jl", lines)
    lines = readlines("Project.toml")
    idx = findfirst(str->startswith(str, "version"), lines)
    lines[idx] = "version = \"0.2.0\""
    writelines("Project.toml", lines)
end
cd(bigpkgpath) do
    lines = readlines("src/BigPkg.jl")
    insert!(lines, 3, "g() = f()")
    writelines("src/BigPkg.jl", lines)
    lines = readlines("Project.toml")
    idx = findfirst(str->startswith(str, "version"), lines)
    lines[idx] = "version = \"0.2.0\""
    lines[end] = "SubPkg = \"0.2\""
    writelines("Project.toml", lines)
    mkdir("test")
    open("test/runtests.jl", "w") do io
        println(io, """
        using BigPkg, Test
        @test BigPkg.g() == 1
        """)
    end

    run(`git add Project.toml src/BigPkg.jl SubPkg/Project.toml SubPkg/src/SubPkg.jl test/runtests.jl`)
    run(`git commit -m "Version 0.2"`)
end

# Remove packages in preparation for simulating what happens when we
# submit a PR and CI runs on Travis
Pkg.rm("SubPkg")
Pkg.rm("BigPkg")

Then I navigate to dev and do this:

tim@diva:/tmp/pkgs/dev$ JULIA_DEPOT_PATH=/tmp/pkgs JL_PKG=BigPkg julia-master --startup-file=no -e 'using Pkg; Pkg.build(verbose=true)'

For some reason, executing the tests as in the default travis script doesn't seem to work for me, but I can do this:

tim@diva:/tmp/pkgs/dev/BigPkg$ JULIA_DEPOT_PATH=/tmp/pkgs julia-master --startup-file=no --project -e 'using Pkg; Pkg.test("BigPkg", coverage=false)'
    Testing BigPkg
┌ Error: Pkg.Resolve.ResolverError("empty intersection between [email protected] and project compatibility 0.2", nothing)
└ @ Pkg.Operations ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1410
ERROR: empty intersection between [email protected] and project compatibility 0.2
Stacktrace:
 [1] resolve_versions!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:370
 [2] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Pkg.Types.UpgradeLevel) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1209
 [3] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:245
 [4] up(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:level, :mode, :update_registry, :io),Tuple{Pkg.Types.UpgradeLevel,Pkg.Types.PackageMode,Bool,Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:68
 [5] resolve(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:251
 [6] (::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec})() at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1412
 [7] with_temp_env(::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec}, ::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1331
 [8] (::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String})(::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1402
 [9] mktempdir(::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String}, ::String; prefix::String) at ./file.jl:682
 [10] mktempdir(::Function, ::String) at ./file.jl:680 (repeats 2 times)
 [11] sandbox(::Function, ::Pkg.Types.Context, ::Pkg.Types.PackageSpec, ::String, ::String, ::Pkg.Types.Project) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1369
 [12] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1533
 [13] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:327
 [14] #test#61 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:67 [inlined]
 [15] #test#60 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:66 [inlined]
 [16] test(::String; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:coverage,),Tuple{Bool}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:65
 [17] top-level scope at none:1
caused by [exception 1]
empty intersection between [email protected] and project compatibility 0.2
Stacktrace:
 [1] resolve_versions!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:370
 [2] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Pkg.Types.UpgradeLevel) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1209
 [3] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:245
 [4] up(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:level, :mode, :update_registry, :io),Tuple{Pkg.Types.UpgradeLevel,Pkg.Types.PackageMode,Bool,Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:68
 [5] resolve(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:251
 [6] resolve(; kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:249
 [7] (::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec})() at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1407
 [8] with_temp_env(::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec}, ::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1331
 [9] (::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String})(::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1402
 [10] mktempdir(::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String}, ::String; prefix::String) at ./file.jl:682
 [11] mktempdir(::Function, ::String) at ./file.jl:680 (repeats 2 times)
 [12] sandbox(::Function, ::Pkg.Types.Context, ::Pkg.Types.PackageSpec, ::String, ::String, ::Pkg.Types.Project) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1369
 [13] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1533
 [14] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:327
 [15] #test#61 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:67 [inlined]
 [16] #test#60 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:66 [inlined]
 [17] test(::String; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:coverage,),Tuple{Bool}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:65
 [18] top-level scope at none:1

So, the obvious problem is that SubPkg is a dependency of BigPkg, and this new version of BigPkg requires [email protected], but [email protected] has not been registered yet.

In #1251, @fredrikekre correctly points out that this is no different from how it works normally. But when you have long dependency chains, how it works normally is really painful. When you have a single git repo, you would like to be able to leverage that organization to develop all the packages in concert. To the developer, the most sensible unit is the repo, not the package.

What do we need? I think the only thing missing is some command that will update the session's notion of what package versions are available by scanning the Project.toml files of all packages listed in the [deps] section of BigPkg. If it does that, and then discovers that in the checked-out copy, lo and behold there is a [email protected], then it can proceed and try everything together.

[timholy]

I'm trying to work around this issue by modifying my ci.yml file (GitHub Actions) so that the relevant section reads

    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
          arch: ${{ matrix.arch }}
      - run: julia -e 'using Pkg; pkg"dev SubPkg"'
      - uses: julia-actions/julia-buildpkg@latest
      - uses: julia-actions/julia-runtest@latest

Sadly, it's not working. Any thoughts? I could go back to adding a build/deps.jl step but that will affect users if I don't forget to strip it out before I ship.

Here's the stacktrace:

ERROR: MethodError: no method matching getindex(::Nothing, ::String)
Stacktrace:
 [1] #build_versions#47(::Bool, ::Function, ::Pkg.Types.Context, ::Array{Base.UUID,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1044
 [2] build_versions at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1033 [inlined]
 [3] #add_or_develop#62(::Array{Base.UUID,1}, ::Symbol, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1204
 [4] #add_or_develop at ./none:0 [inlined]
 [5] #add_or_develop#15(::Symbol, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:69
 [6] (::getfield(Pkg.API, Symbol("#kw##add_or_develop")))(::NamedTuple{(:mode,),Tuple{Symbol}}, ::typeof(Pkg.API.add_or_develop), ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at ./none:0
 [7] do_develop!(::Dict{Symbol,Any}, ::Array{Pkg.Types.PackageSpec,1}, ::Dict{Symbol,Any}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:714
 [8] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:697
 [9] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:696
 [10] do_cmd!(::Pkg.REPLMode.PkgCommand, ::Pkg.REPLMode.MiniREPL) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:603
 [11] #do_cmd#33(::Bool, ::Function, ::Pkg.REPLMode.MiniREPL, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:577
 [12] (::getfield(Pkg.REPLMode, Symbol("#kw##do_cmd")))(::NamedTuple{(:do_rethrow,),Tuple{Bool}}, ::typeof(Pkg.REPLMode.do_cmd), ::Pkg.REPLMode.MiniREPL, ::String) at ./none:0
 [13] top-level scope at none:0

[KristofferC]

The code to repro this seems to not work 100%? It gave me a Project.toml that looked like

name = "BigPkg"
uuid = "e4dbb02c-8206-4006-8b08-6815cbd025b8"
SubPkg = "0.2"

I guess it should be something like

name = "BigPkg"
uuid = "e4dbb02c-8206-4006-8b08-6815cbd025b8"

[deps]
SubPkg = "acd...."

[compat]
SubPkg = "0.2"

Couldn't you check in a manifest in BigPkg where all subdir dependencies are devved? That way you always run CI on the "state of the repo". You could run Pkg.update before testing if you want to use the last version of the other dependencies.

[timholy]

Yes, that looks borked. Strange, it worked perfectly for me. I just tested again, and again it was fine.

The Manifest solution seems to work, many thanks for suggesting what should have been obvious! I have actively avoided them for various reasons (worries about testing stale rather than latest versions of pkgs, concerns about how well they generalize across Julia versions, etc), but this does sound like an excellent workaround. I think we still might consider not making an Manifest.toml obligate, but for the time being this is a simple solution.

[timholy]

Is it possible to add multiple Manifest files for different Julia versions? We're getting CI failures due to what appear to be Julia-version [compat] requirements of some of our dependencies. Yes, we could manually pick an earlier version, but it seems likely that there will be some packages for which there is no single version that works across all Julia versions.

[StefanKarpinski]

There's no way to do that currently, and I'm not sure we want to go down that path. I could turn out to be wrong but it feels like one of those annoyances that may sort itself out over time if we suffer through the discomfort for a while and figure out other ways to deal with it. If you're running code against multiple different Julia versions, isn't it better to not check the manifest in and let the resolver pick versions that are compatible with your Julia version?

[timholy]

Currently that's the only way I know of, other than adding a deps/build.jl script that devs subdirectory-packages (which is a worse solution), of ~solving~circumventing this issue. But I'd rather not check in the Manifest.toml at all, frankly.

[KristofferC]

You could dev the sub-packages before running the tests in CI perhaps?

[aminya]

You could dev the sub-packages before running the tests in CI perhaps?

I am confused a bit. This is similar to https://github.com/timholy/SnoopCompile.jl/commit/bb56316297de70d401e84433bf84064cd9cee9fe and my whole issue on it https://github.com/JuliaLang/Pkg.jl/issues/1828.

If running this script is a bad thing we should not do it at all (including in CI). If it is a good thing why we don't do it by default.

[fredrikekre]

You could dev the sub-packages before running the tests in CI perhaps?

I am confused a bit. This is similar to timholy/SnoopCompile.jl@bb56316 and my whole issue on it #1828

No, the dev suggestion is part of installing the package, not some runtime adding of packages.

[aminya]

Both suggestions that I made were are also meant for the installation phase of a package.

[fredrikekre]

No they would be executed after installation. The build script runs after all packages have been resolved. You should not run Pkg operations in the build script and https://github.com/timholy/SnoopCompile.jl/commit/bb56316 is invalid.

[timholy]

Worth mentioning that trying to do this from the ci.yml file is not yet working. Any ideas? https://github.com/timholy/SnoopCompile.jl/pull/107

[aminya]

Worth mentioning that trying to do this from the ci.yml file is not yet working. Any ideas? timholy/SnoopCompile.jl#107

That is probably the Powershell issue I gave a solution for here.

[aminya]

No they would be executed after installation. The build script runs after all packages have been resolved. You should not run Pkg operations in the build script and timholy/SnoopCompile.jl@bb56316 is valid.

I have no objection to running it before doing any Pkg operation. Project.jl could run before installation. Here, I abused deps/build.jl as the only possible way of installing something, as I mentioned in https://github.com/JuliaLang/Pkg.jl/issues/1828#issuecomment-632470085

You could dev the sub-packages before running the tests in CI perhaps?

Now, I need to dev these packages everywhere that I want to test a developing SnoopCompile branch. Pkg.add simply does not work. See
https://github.com/timholy/SnoopCompile.jl/pull/107#issuecomment-649028693

[fredrikekre]

Now, I need to dev these packages everywhere that I want to test a developing SnoopCompile branch.

No, this is only because those packages are not registered yet. When they are you only have to dev SnoopCompile to develop SnoopCompile, just like you only need to dev DataFrames to develop DataFrames.

[KristofferC]

I must say that even with this dedicated issue I have a bit of a hard time wrapping my head about what needs to be done and what

I think the only thing missing is some command that will update the session's notion of what package versions are available by scanning the Project.toml files of all packages listed in the [deps] section of BigPkg. If it does that, and then discovers that in the checked-out copy, lo and behold there is a [email protected], then it can proceed and try everything together.

actually means. This is literally what happens when a package is developed and you run resolve so I don't understand why that is not the solution. If you want to track packages by local files you use dev, if you want to track packages by registered versions you use add.

[timholy]

(@v1.6) pkg> st SnoopCompile SnoopCompileCore SnoopCompileAnalysis SnoopCompileBot
Status `~/.julia/environments/v1.6/Project.toml`
  [aa65fe97] SnoopCompile v1.6.0
  [9ea4277c] SnoopCompileAnalysis v1.6.0
  [1d5e0e55] SnoopCompileBot v1.6.0
  [e2b509da] SnoopCompileCore v1.6.0

(@v1.6) pkg> dev SnoopCompile
Path `/home/tim/.julia/dev/SnoopCompile` exists and looks like the correct package. Using existing path.
  Resolving package versions...
ERROR: Unsatisfiable requirements detected for package SnoopCompileBot [1d5e0e55]:
 SnoopCompileBot [1d5e0e55] log:
 ├─possible versions are: 1.6.0 or uninstalled
 └─restricted to versions 1.6.1-1.6 by SnoopCompile [aa65fe97] — no versions left
   └─SnoopCompile [aa65fe97] log:
     ├─possible versions are: 1.6.1 or uninstalled
     └─SnoopCompile [aa65fe97] is fixed to version 1.6.1

(@v1.6) pkg> resolve
No Changes to `~/.julia/environments/v1.6/Project.toml`
No Changes to `~/.julia/environments/v1.6/Manifest.toml`

(@v1.6) pkg> dev SnoopCompile
Path `/home/tim/.julia/dev/SnoopCompile` exists and looks like the correct package. Using existing path.
  Resolving package versions...
ERROR: Unsatisfiable requirements detected for package SnoopCompileBot [1d5e0e55]:
 SnoopCompileBot [1d5e0e55] log:
 ├─possible versions are: 1.6.0 or uninstalled
 └─restricted to versions 1.6.1-1.6 by SnoopCompile [aa65fe97] — no versions left
   └─SnoopCompile [aa65fe97] log:
     ├─possible versions are: 1.6.1 or uninstalled
     └─SnoopCompile [aa65fe97] is fixed to version 1.6.1

(@v1.6) pkg> dev SnoopCompile SnoopCompileCore SnoopCompileAnalysis SnoopCompileBot
[ Info: Resolving package identifier `SnoopCompileCore` as a directory at `~/.julia/dev/SnoopCompile/SnoopCompileCore`.
[ Info: Resolving package identifier `SnoopCompileAnalysis` as a directory at `~/.julia/dev/SnoopCompile/SnoopCompileAnalysis`.
[ Info: Resolving package identifier `SnoopCompileBot` as a directory at `~/.julia/dev/SnoopCompile/SnoopCompileBot`.
Path `/home/tim/.julia/dev/SnoopCompile` exists and looks like the correct package. Using existing path.
Path `SnoopCompileCore` exists and looks like the correct package. Using existing path.
Path `SnoopCompileAnalysis` exists and looks like the correct package. Using existing path.
Path `SnoopCompileBot` exists and looks like the correct package. Using existing path.
  Resolving package versions...
Updating `~/.julia/environments/v1.6/Project.toml`
  [aa65fe97] ~ SnoopCompile v1.6.0 ⇒ v1.6.1 `~/.julia/dev/SnoopCompile`
  [9ea4277c] ~ SnoopCompileAnalysis v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileAnalysis`
  [1d5e0e55] ~ SnoopCompileBot v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileBot`
  [e2b509da] ~ SnoopCompileCore v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileCore`
Updating `~/.julia/environments/v1.6/Manifest.toml`
  [aa65fe97] ~ SnoopCompile v1.6.0 ⇒ v1.6.1 `~/.julia/dev/SnoopCompile`
  [9ea4277c] ~ SnoopCompileAnalysis v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileAnalysis`
  [1d5e0e55] ~ SnoopCompileBot v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileBot`
  [e2b509da] ~ SnoopCompileCore v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileCore`

I'm not sure how it would be implemented, but the above is not ideal.

[johnnychen94]

Am I understand it wrongly, or a Pkg.update() seems already a good solution to this?

- JULIA_DEPOT_PATH=/tmp/pkgs julia-latest --startup-file=no --project -e 'using Pkg; Pkg.test("BigPkg", coverage=false)'
+ JULIA_DEPOT_PATH=/tmp/pkgs julia-latest --startup-file=no --project -e 'using Pkg; Pkg.update(); Pkg.test("BigPkg", coverage=false)'

jc@mathlf1:/tmp/pkgs/dev/BigPkg$ J ULIA_DEPOT_PATH=/tmp/pkgs julia-latest --startup-file=no --project -e 'using Pkg; Pkg.update(); Pkg.test("BigPkg", coverage=false)'

   Updating registry at `/tmp/pkgs/registries/General`
   Updating registry at `/tmp/pkgs/registries/MyRegistry`
   Updating git-repo `/tmp/pkgs/registries/MyRegistry`
   Updating git-repo `/tmp/pkgs/dev/BigPkg`
Updating `/tmp/pkgs/dev/BigPkg/Project.toml`
  [352b57a5] ~ SubPkg v0.1.0 `.:SubPkg#master` ⇒ v0.2.0 `.:SubPkg#master`
Updating `/tmp/pkgs/dev/BigPkg/Manifest.toml`
  [352b57a5] ~ SubPkg v0.1.0 `.:SubPkg#master` ⇒ v0.2.0 `.:SubPkg#master`
    Testing BigPkg
Status `/tmp/jl_zZBmcy/Project.toml`
  [5abf6831] BigPkg v0.2.0 `/tmp/pkgs/dev/BigPkg`
  [352b57a5] SubPkg v0.2.0 `.:SubPkg#master`
Status `/tmp/jl_zZBmcy/Manifest.toml`
  [5abf6831] BigPkg v0.2.0 `/tmp/pkgs/dev/BigPkg`
  [352b57a5] SubPkg v0.2.0 `.:SubPkg#master`
ERROR: LoadError: ArgumentError: Package Test not found in current path:
- Run `import Pkg; Pkg.add("Test")` to install the Test package.

Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:893
 [2] include(::String) at ./client.jl:444
 [3] top-level scope at none:6
in expression starting at /tmp/pkgs/dev/BigPkg/test/runtests.jl:1
ERROR: Package BigPkg errored during testing
Stacktrace:
 [1] pkgerror(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:52
 [2] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1561
 [3] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:327
 [4] #test#61 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:67 [inlined]
 [5] #test#60 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:66 [inlined]
 [6] test(::String; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:coverage,),Tuple{Bool}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:65

[KristofferC]

I'm not sure how it would be implemented, but the above is not ideal.

What's the not ideal part? That you have to dev them in one command?

[johnnychen94]

My understanding is that when you dev a package, it should automatically dev all sub-packages since they're in one big git repo. It doesn't make much sense to still use the released version of those sub-packages.

Take SnoopCompile as the very specific case:

• (@v1.6) pkg> dev SnoopCompile: also devs SnoopCompile, SnoopCompileCore, SnoopCompileAnalysis, SnoopCompileBot
• (@v1.6) pkg> dev SnoopCompileBot: also devs SnoopCompileCore and SnoopCompileAnalysis
• (@v1.6) pkg> dev SnoopCompileCore: as it is

[aminya]

Yes. I made a dev.jl for my personal use to do that.
https://github.com/aminya/SnoopCompile.jl/blob/devscript/dev.jl

[KristofferC]

My understanding is that when you dev a package, it should automatically dev all sub-packages since they're in one big git repo. It doesn't make much sense to still use the released version of those sub-packages.

Just to point out, there is no concept (right now at least) of a sub-package. There are just packages and they can now be stored in the same git-repo. It is not clear at all to me that automatically devving all packages that are in the same repo just because you dev one of them is something you would always want.

Yes. I made a dev.jl for my personal use to do that.

Would be better to do all the devs in one command (develop can take a vector of PackageSpec).

[timholy]

Deciding specifically what is wrong is definitely a gray zone. Let me try in several steps:

• nothing is documented about this, everyone has to figure it out on their own. https://github.com/timholy/SnoopCompile.jl/pull/107 did not exactly happen automatically without effort. At a bare minimum this needs to be documented. I've edited the top post here to reflect that, but if that's all we end up doing then it needs to go somewhere more permanent.
• it's not completely intuitive. It might be to people who wrote Pkg (I am eternally grateful), but I think many of us want to think of one repo as an "uber-package." Really, the feature I think I really want (but don't have) is the ability to load specific sub-modules, in which case this really would be just one package. But that doesn't seem to be on the horizon (and it's not entirely trivial to see how it would work), and this already is 98% working, so I'm interested in exploring whether we can smooth over that last 2%. Yes, that last 2% might be a corruption of Pkg's design. I think we need to think about it anyway---the current workflow for large ecosystems is too painful. (That's not a new problem, but now we almost have a fix.)

It seems to me that we need some way of saying, "this collection of packages belongs together." Maybe something like a new

[subpkgs]
SubPkgA
SubPkgB
SubPkgC/SubPkgCa
SubPkgC/SubPkgCb

section to the "umbrella" Project.toml. Pkg looks for that and then does some stuff differently.

But I'm open to other solutions. I definitely don't want to create src-chaos if there's an easier way to smooth the workflow.

[timholy]

It is not clear at all to me that automatically devving all packages that are in the same repo just because you dev one of them is something you would always want.

I strongly agree with you there. This should be something you opt into. I'm not certain how flexible this needs to be? Should it be "all one or none"? Or would people want to group some packages but not others? E.g.,

[subpkgs]
SnoopCompile = ["SubPkgB", "SubPkgC/SubPkgCa", "SubPkgC/SubPkgCb"]
SomeOtherTool = ["ItsSubPkgA", ...]

[timholy]

Let me add one more problem, automerge doesn't work: https://github.com/JuliaRegistries/General/pull/16914. Can someone merge it after waiting 15 minutes (if no comments posted)?

[DilumAluthge]

Let me add one more problem, automerge doesn't work: JuliaRegistries/General#16914. Can someone merge it after waiting 15 minutes (if no comments posted)?

If someone can add support to Registrator.jl for registering multiple packages/versions at the same time, then we can work on adding support to AutoMerge at well. Basically, Registrator would need to give us a list of package-version pairs(instead of a single package-version pair) and we would loop over that list and run the automerge checks for each package-version pair.

@timholy Can you open an issue on RegistryCI.jl to track support for multiple registrations?

[timholy]

https://github.com/JuliaRegistries/RegistryCI.jl/issues/256

[KristofferC]

nothing is documented about this, everyone has to figure it out on their own

About what though? If there was a difference between how things worked with multiple packages inside a repo or not then I agree. However, things now work exactly the same as they always have so what is there to document? The addition of subdir support to Pkg was a couple of lines so it isn't like anything major happened from it. We could add a workflow tip that if you have multiple packages in the same repo and want to run tests with the local content of all these packages, then you should dev all the dependencies in the repo before running the tests.

It seems to me that we need some way of saying, "this collection of packages belongs together."

Really, the feature I think I really want (but don't have) is the ability to load specific sub-modules, in which case this really would be just one package.

I think we need to think about it anyway---the current workflow for large ecosystems is too painful. (That's not a new problem, but now we almost have a fix.)

The main purpose of subdir support was so that people could add a package to a repo where the julia package was not the most important part. For example, I think the Arrow repository has packages for many programming languages inside the repo, but previously it wasn't possible to add a Julia package there since it had to be at toplevel. It was never really intended for people to start cutting up their package into a bunch of "sub-packages" (since we don't have a concept for that). Although if you do that, the situation now is the same as it was before if you would have each of these sub-packages in a separate repo.

[DilumAluthge]

It was never really intended for people to start cutting up their package into a bunch of "sub-packages" (since we don't have a concept for that).

For what it's worth, when I opened https://github.com/JuliaLang/Pkg.jl/issues/1251, that was actually one of the use cases that I had in mind: breaking up big packages into many smaller packages.

Although if you do that, the situation now is the same as it was before if you would have each of these sub-packages in a separate repo.

I think Tim and I agree with you that currently it makes no difference whether these "multiple little packages" live in the same repo or separate repos. But I think what Tim is getting at is that now that we can put all these packages into a single repo, we have the potential to make this situation easier in the future. That is, putting all these packages into a single repo has the potential to be strictly easier than keeping them in separate repos. We just need to implement the functionality (wherever it needs to go) to achieve this "strictly easier" state.

[timholy]

It was never really intended for people to start cutting up their package into a bunch of "sub-packages" (since we don't have a concept for that).

While SnoopCompile is indeed an example of that, the main use I have in mind is the reverse: I want to take about 20 individual packages and recombine them into a single repo, and then "lock" them together so they are a unit, while still allowing users to load individual (sub)packages. All of this is about managing Julia's latency. If latency weren't an issue, these would all be a single package, end of story, and this never would have come up.

So this conversation isn't so confusing, I propose we change it: let's start designing a mechanism to declare that a single package produces multiple *.ji files that can be individually loaded. What changes to the code-loading & Pkg internals do you envision? Now think of this issue as arriving at what is functionally a similar point, but starting from multiple packages. What is needed to turn the latter into the equivalent of the former?

Does that help? EDIT: and of course, I'm happy to have that alternative conversation, particularly if you think it's a better way to solve the problem.

[KristofferC]

declare that a single package produces multiple *.ji files that can be individually loaded.

Okay, so let's start thinking about that.

There is one package with one version, one set of dependencies, one set of compat info, you can only depend on the "full package", the only difference is that you can load a submodule of the package and have the submodule compile to a ji file in order to selectively load a part of a package to reduce latency.

What we need:

• Syntax to load the subpackage: let's say: using MainPackage@SubPackage.
• A file layout convention, let's say we make it: src/MainPackage.jl and src/SubPackage.jl.
• A name schema for the ji file, let's call it: $MAINPACKAGE_JI * _ * "SubPackage.ji" where MAINPACKAGE_JI is the name of the ji file for the main package.

Now, writing using MainPackage@SubPackage would:

• Look up the UUID / version for MainPackage, and find the path to it.
• Instead of loading src/MainPackage.jl and running the whole precompilation story, it would do the same for src/SubPackage.jl.

Note that the source of src/MainPackage.jl would then not be

module MainPackage
include("SubPackage.jl")
end

but more something like

module MainPackage
import MainPackage@SubPackage
end

I don't think much more than that has to be done. No change in Project files nor Pkg.

The drawbacks:

• If you have HugePackage.jl with a subpackage MiniPackage.jl you need to download all dependencies of HugePackage even though you only want to use MiniPackage.jl.
• If anything is incompatible with HugePackage.ji, the resolver will complain, even if MiniPackage.jl is fine to install on its own.
• A breaking change to SubPackage2.jl will also cause a new breaking version to be tagged for SubPackage1.jl even though nothing changed in that pacakge.

[timholy]

Yes! This! And I'm fine with all those drawbacks. Many of them would indeed be a relief---it's really nice to say "this part of your environment has priority."

[timholy]

One issue, though: if HugePackage depends on 100 other packages, but HugePackage@MiniPackage only needs 3 of them, does using HugePackage@MiniPackage just load the 3?

[KristofferC]

does using HugePackage@MiniPackage just load the 3?

Yes, since it will load src/MiniPackage.jl which does only 3 package loads. The concrete set of dependencies is only stored in the Manifest.toml and is only used in code loading to look them up when they happen to be needed.

[timholy]

I'm confused about how the appropriate details for HugePackage@MiniPackage's dependencies get declared. I guess the [deps] section of HugePackage/Project.toml is just a name-resolver and the actual chain of dependencies is determined by the code-loading machinery? I guess I could test that by introducing a spurious dependency and then seeing if it shows up in the Manifest.toml, but maybe you know the answer.

[KristofferC]

I'm confused about how the appropriate details for HugePackage@MiniPackage's dependencies get declared.

It would be free to load any dependency of HugePackage.

I guess the [deps] section of HugePackage/Project.toml is just a name-resolver and the actual chain of dependencies is determined by the code-loading machinery?

Yes, pretty much.

I guess I could test that by introducing a spurious dependency and then seeing if it shows up in the Manifest.toml, but maybe you know the answer.

Don't really understand what this means.

[timholy]

Oh, wait, now perhaps I am beginning to see: there wouldn't be any Manifest entry for HugePackage@Minipackage, just for HugePackage. And then it gets used by the name-resolution machinery during code-loading. Duh, I should have known that.

[KristofferC]

Oh, wait, now perhaps I am beginning to see: there wouldn't be any Manifest entry for HugePackage@Minipackage, just for HugePackage.

Yes, this is accurate.

[KristofferC]

It should be stated that the more that HugePackge gets cut up, the slower it is likely to load (ref https://github.com/JuliaGraphics/Gtk.jl/issues/466).

[timholy]

Warning noted. But again, in the case I really care about, I've already done the cutting up (all the way back in the Julia 0.x for small x days). I want to glue them back together.

[johnnychen94]

Does this mean Minipackage doesn't need to be registered in the General registry anymore? If so there could be some confusion between using Minipackage and using HugePackage@minipackage, especially when the original Minipackage repo is deprecated and not updated.

How about package tests? What would happen when ] test HugePackage, does it include all tests in those Minipackages? will ] test HugePackage@Minipackage work?

[GunnarFarneback]

This was a too long read to go into all details, so please excuse me if I've missed or misunderstood something important.

I think the layout

BigPkg/
├── A
│   └── Project.toml
├── B
│   └── Project.toml
├── C
│   └── Project.toml
└── Project.toml

where BigPkg is an umbrella package for A, B, C, is the wrong model for this. I think it should rather be

├── A
│   └── Project.toml
├── B
│   └── Project.toml
├── BigPkg
│   └── Project.toml
├── C
│   └── Project.toml
├── Manifest.toml
└── Project.toml

where the top level Project.toml and Manifest.toml permanently devs all the packages in the repo. Then you would run CI from the top level repo and if it's fine, bump the package versions and register them all together. Obviously there's bit of tooling missing for the latter steps but I would be happy to host experiments on this in LocalRegistry, which would be a more convenient playground than Registrator.

(I have no personal need of this. My interest in subdir packages has always been on the case where the Julia package is only a smaller part of a repo.)

[ma-laforge]

It is not clear at all to me that automatically devving all packages that are in the same repo just because you dev one of them is something you would always want.

I strongly agree with you there. This should be something you opt into. I'm not certain how flexible this needs to be? Should it be "all one or none"? Or would people want to group some packages but not others? E.g.,

[subpkgs]
SnoopCompile = ["SubPkgB", "SubPkgC/SubPkgCa", "SubPkgC/SubPkgCb"]
SomeOtherTool = ["ItsSubPkgA", ...]

I think I disagree with you here, @KristofferC & @timholy. If you create a multi-package repo in a single .git repository, I think this sort of makes sense.

dev-ing "affects" all sub-packages

On a purely practical standpoint, you must check out the entire repo when you try to dev one of the subpackages. So really, I think dev RootPkg should trigger devving of all sub-packages in that repo -- and have all subsequent calls to import RootPkg@SubPkg* redirect to the dev-ed version of the package.

It is the most natural, orthogonal way to do development, I think.

Repo as an organizational unit

In most cases, package developers should use the .git repo as a means to collect somewhat related packages/functionality only.

If we want to support the possibility for developers dump all kinds of unrelated packages in a single repo... well, that's fine. However, I really think the primary goal should be to have a simple & efficient workflow to deal with the former case, though.

IMPORTANT: I wish to promote the concept of sub-packages instead of using our present solution, as is. If we can develop a workflow with the sub-package concept in mind, I believe it can vastly improve the development of complex multi-package solutions within a single .git repo.

Use case: adding a new feature

Let's assume we bundled Plots.jl and its supporting packages into a single .git repo. When I eventually want to add a new feature to Plots, there is a good chance I'll also have to add code to to RecipesPipeline & RecipesBase in order to address the solution correctly.

#Plots.jl.git repo could contain all its related dependencies!:
Plots -> RecipesPipeline -> RecipesBase
         PlotUtils
         PlotThemes -> PlotUtils
         RecipesBase
         RecipesPipeline -> RecipesBase
                            PlotUtils

So yes: I think it makes tons of sense to have all sub-packages deved with a call to dev Plots.

Package-level micro-registries

To me, it makes the most sense to have package-level micro-registries. In theory, you should only need to register the root package (RootPkg) with JuliaRegistries/General. When you register (RootPkg), the registration tools should be able to parse the its Project.toml file in order to find all sub-packages:

name = "RootPkg"
uuid = "aa65fe97-06da-5843-b5b1-d5d13cad87d2"
version = "1.7.1"

[subpkgs]
SubPkgB = "subpkgs/SubPkgB"
SubPkgCa = "subpkgs/SubPkgC/SubPkgCa"
SubPkgCb = "subpkgs/SubPkgC/SubPkgCb"

[deps]
SubPkgB = "9ea4277c-da97-4c3a-afb0-537c066769de"
SubPkgCa = "1d5e0e55-7d74-4714-b8d8-efa80e938cf7"
SubPkgCb = "e2b509da-e806-4183-be48-004708413034"

Keeping versions together
What is key here is that these sub-packages should not typically need to be versioned In the sub-package-capable workflow. They should basically inherit the version number from RootPkg.

[ma-laforge]

@KristofferC What we need:

- [...]
- A file layout convention, let's say we make it: `src/MainPackage.jl` and `src/SubPackage.jl`.
- [...]

I like the simplicity of this file layout, but this might not be a practical way to declare the dependencies for MainPackage@SubPackage. If we instead place sub-packages in their own sub-directories, we can bundle in their own Project.toml file - thus allowing the sub-packages to have fewer dependencies.

Subpackages with MORE dependencies

If I were to add another requirement: it would be nice if certain subpackages could also have more dependencies than the root package. For example:

SnoopCompile [dep: Pkg, SnoopCompile@SnoopCompileCore, SnoopCompile@SnoopCompileAnalysis, ...]
    ->SnoopCompileCore [dep: Serialization]
    ->SnoopCompileAnalysis [dep: Cthulhu, OrderedCollections, Serialization]
    ->SnoopCompileBot [dep: Pkg, FilePathsBase, SnoopCompile@SnoopCompileCore, ...]

#Additional/Wrapper packages:
    ->SnoopCompileGUI [dep: SnoopCompile, Gtk, ...]

In the future, it might make sense to bundle a GUI to aid in using SnoopCompile. If we include it as a sub-project with additional dependencies, it means we don't have to add Gtk.jl to all Julia environments that want to use SnoopCompile.

Well, maybe a GUI for SnoopCompile wouldn't really be appropriate, but I hope you get the idea.

[ma-laforge]

@GunnarFarneback where BigPkg is an umbrella package for A, B, C, is the wrong model for this. I think it should rather be

├── A
│   └── Project.toml
├── B
│   └── Project.toml
├── BigPkg
│   └── Project.toml
├── C
│   └── Project.toml
├── Manifest.toml
└── Project.toml

Before answering, I would like to rename Repo->RootPkg to clarify the discussion. That way, you can have BOTH a RootPkg, and a BigPkg.

What I like about this topology is that RootPkg.Project.toml can be created have zero dependencies on its own. In turn, that would mean that:

• ] add RootPkg@A could pull in ONLY the direct dependencies of A.
• Most people will probably ] add RootPkg@BigPkg: Get all functionality from BigPkg.

I really like this idea. The only problem is that it is slightly less intuitive than @timholy 's original proposal where RootPkg=BigPkg. But that's just because you no longer get the implicit positional hint that all subpackages provide the functionality of RootPkg=BigPkg. Despite this small drawback, I think this solution is more flexible and appropriate.

[GunnarFarneback]

Before answering, I would like to rename Repo->RootPkg

I intentionally chose a non-package name for the root of the repo because

1. I'm not convinced that there really is a need for the subpackage concept. It looks more like a tooling problem to me.
2. Unless there is a major redesign of Pkg, a root package will contain the contents of all packages in subdirectories, as well as non-package subdirectories if there are any. I'm not sure everybody is aware of this.

[ma-laforge]

• I'm not convinced that there really is a need for the subpackage concept. It looks more like a tooling problem to me.

I'm not sure I understand. Yes, I think it is a tooling problem: the package/registration system lacks the ability to deal with multiple packages in the same .git repo in a way that is practical for development, testing, and keeping dependencies to a minimum. The subpackage concept is simply an abstraction we use to describe our problem/solution.

To me, this feels alot like the one file per function paradigm in Matlab. Yes, you can just bite your tongue and break everything into singe files, but you tend to get lost in your files, and development time suffers.

Actually, it is worse than the one file per function issue, because it causes long delays in creating releases. Even the part of "push" ing each individual .git repo/julia package to Github is a pain when your solution is broken down into so many repositories.

• Unless there is a major redesign of Pkg, a root package will contain the contents of all packages in subdirectories, as well as non-package subdirectories if there are any. I'm not sure everybody is aware of this.

Yes. A package was originally defined at the .git repo level. So yes, when you add that package, Julia pulls the entire repo. I assume most people know this. The decision to tie packages to a .git repo seems to be the core reason why there is a need for the sub-package concept.

And I agree, this might require a relatively major redesign of Pkg. If it can be done in an effective way by adding support for subdirectories, as well as a few registration scripts, then that would be excellent! Sadly, I'm not convinced that's all that we need to make this aspect of package development user-friendly.

[GunnarFarneback]

And I agree, this might require a relatively major redesign of Pkg

Well, that's where we disagree. I don't think Pkg needs (more) changes to effectively support multiple packages in one repository. I just think you shouldn't try to place a package at the root of the repository if it includes multiple packages.

[DilumAluthge]

Yeah, when I first suggested this feature, I did not imagine that one package would be in a subdirectory of another.

[ma-laforge]

Well, that's where we disagree. I don't think Pkg needs (more) changes to effectively support multiple packages in one repository.

If "effectively" is the operative word here, I think the difficulty @timholy is having to get SnoopCompile working in a multi-package context says otherwise. Maybe Pkg doesn't need more changes, but something probably does. I just don't know what that thing is at this time.

I just think you shouldn't try to place a package at the root of the repository if it includes multiple packages.

I have no strong opinion on this. In this context, I'm completely fine with there not being a package at the root of my repositories. This restriction doesn't really pose an obstacle for my particular use cases. In fact, it might end up being less confusing to add this restriction.

[johnnychen94]

The simplest package development workflow is to have one big package for all related functions. As @ma-laforge said, developing one big repo makes version control less a headache, especially when the change involves multiple packages. A gripe is that this giant package makes precompilation time a big issue for both developers and users.

I wanted to echo @timholy that the concept of sub-packages, IMO, gives us the best of two worlds that 1) it's easier to make a change to multiple sub-packages without propagating incompleteness to downstream packages 2) it doesn't explode the precompilation time.

[KristofferC]

FWIW, I am very much losing the overview of this issue, what is proposed, what is considered bug reports, what is considered feature requests, what is just random discussion from the SnoopCompile split etc. I'm going to close this, so please open a new issue with a concrete descriptive proposal.

[GunnarFarneback]

Let's go back to the beginning and apply the layout from https://github.com/JuliaLang/Pkg.jl/issues/1874#issuecomment-653762205 to the reproducer. Then the generate_test.jl would be changed to

function writelines(path, lines)
    open(path, "w") do io
        for line in lines
            println(io, line)
        end
    end
end

# Create a Root repository with subdir packages BigPkg and SubPkg
# Make BigPkg depend on SubPkg
# Register them
using Pkg
Pkg.add("LocalRegistry")
using LocalRegistry
regpath = joinpath(@__DIR__, "registries", "MyRegistry")
create_registry("MyRegistry", regpath)
# Pkg.Registry.add(regpath)
devpath = joinpath(@__DIR__, "dev")
mkpath(devpath)
rootpath = joinpath(devpath, "Root")
bigpkgpath = joinpath(rootpath, "BigPkg")
subpkgpath = joinpath(rootpath, "SubPkg")
Pkg.generate(bigpkgpath)
Pkg.generate(subpkgpath)
Pkg.activate(rootpath)
Pkg.develop(path=bigpkgpath)
Pkg.develop(path=subpkgpath)
cd(rootpath) do
    run(`git init`)
    run(`git add Project.toml Manifest.toml BigPkg/Project.toml BigPkg/src/BigPkg.jl SubPkg/Project.toml SubPkg/src/SubPkg.jl`)
    run(`git commit -m "Skeleton"`)

    Pkg.activate("BigPkg")
    Pkg.add("Test")
    Pkg.add(path=".", subdir="SubPkg")
    lines = readlines("BigPkg/Project.toml")
    append!(lines, ["", "[compat]", "SubPkg = \"0.1\""])
    writelines("BigPkg/Project.toml", lines)
    lines = readlines("BigPkg/src/BigPkg.jl")
    insert!(lines, 2, "using SubPkg")
    writelines("BigPkg/src/BigPkg.jl", lines)
    Pkg.activate(rootpath)
    Pkg.resolve()
    run(`git add BigPkg/Project.toml BigPkg/src/BigPkg.jl`)
    run(`git commit -m "Add SubPkg dependency"`)
end
using BigPkg
using SubPkg
register(SubPkg, "MyRegistry"; repo=bigpkgpath)
register(BigPkg, "MyRegistry"; repo=bigpkgpath)

# OK, now both packages are registered at v0.1.
# Let's add a new feature to SubPkg and bump its version
# to 0.2. Simultaneously, let's require the new feature
# in BigPkg and bump its version to 0.2.
cd(subpkgpath) do
    lines = readlines("src/SubPkg.jl")
    insert!(lines, 2, "export f")
    insert!(lines, 3, "f() = 1")
    writelines("src/SubPkg.jl", lines)
    lines = readlines("Project.toml")
    idx = findfirst(str->startswith(str, "version"), lines)
    lines[idx] = "version = \"0.2.0\""
    writelines("Project.toml", lines)
end
cd(bigpkgpath) do
    lines = readlines("src/BigPkg.jl")
    insert!(lines, 3, "g() = f()")
    writelines("src/BigPkg.jl", lines)
    lines = readlines("Project.toml")
    idx = findfirst(str->startswith(str, "version"), lines)
    lines[idx] = "version = \"0.2.0\""
    lines[end] = "SubPkg = \"0.2\""
    writelines("Project.toml", lines)
    mkdir("test")
    open("test/runtests.jl", "w") do io
        println(io, """
        using BigPkg, Test
        @test BigPkg.g() == 1
        """)
    end

    run(`git add Project.toml src/BigPkg.jl ../SubPkg/Project.toml ../SubPkg/src/SubPkg.jl test/runtests.jl`)
    run(`git commit -m "Version 0.2"`)
end

Performing the same steps as in the issue description, I get

gunnar@ubuntu:/tmp/pkgs/dev/Root$ JULIA_DEPOT_PATH=/tmp/pkgs ~/julia1.5/julia --startup-file=no --project -e 'using Pkg; Pkg.test("BigPkg", coverage=false)'
    Testing BigPkg
Status `/tmp/jl_9a76kQ/Project.toml`
  [5f1f962e] BigPkg v0.2.0 `/tmp/pkgs/dev/Root/BigPkg`
  [d729db4a] SubPkg v0.2.0 `/tmp/pkgs/dev/Root/SubPkg`
  [8dfed614] Test
Status `/tmp/jl_9a76kQ/Manifest.toml`
  [5f1f962e] BigPkg v0.2.0 `/tmp/pkgs/dev/Root/BigPkg`
  [d729db4a] SubPkg v0.2.0 `/tmp/pkgs/dev/Root/SubPkg`
  [2a0f44e3] Base64
  [8ba89e20] Distributed
  [b77e0a4c] InteractiveUtils
  [56ddb016] Logging
  [d6f4376e] Markdown
  [9a3f8284] Random
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [8dfed614] Test
    Testing BigPkg tests passed 

@timholy, is this enough to solve this part of the puzzle?

[StefanKarpinski]

With apologies to @timholy, when this got to "relatively major redesign of Pkg" I think we lost the thread here. Let's try again with some more focused issues. If folks want to continue the larger brainstorming here, feel free.

[ma-laforge]

I apologize as well. When I said "this might require a relatively major redesign of Pkg", I actually meant I expect the changes won't be a couple of non-trivial one liners.

Dealing with multi-package repositories in a way that is easy for development, unit testing, and publication/registration requires a bit of thought. This is especially true if we want to maintain minimum sub-package dependencies to avoid pulling in more packages than a project requires.

However, for the most part, I think the package system does an excellent job dealing with very complex issues.

[StefanKarpinski]

This issue has come up again on Slack and the fact that I "hated it" was mentioned as the reason for abandoning the idea. To be clear: I would be totally fine with a feature that allowed import A.B to load B without having to load all of the rest of A. However, such a feature should

1. not introduce any new concepts that the caller needs to understand
2. not introduce any new syntax so that the caller can just do import A.B and not be aware that B is special

The proposals being discussed here violate both of those and introduces a subtly different new concept—so subtle that from the user's perspective it's identical to a submodule, it only different from the developer's perspective, which seems not great, Bob. It also introduces a new syntax which, so that the user has to guess which of import A.B or import A@B they're supposed to write. Since there's no user-visible difference in the effect of these, it's a pure guessing game which one they're supposed to write.

[KristofferC]

The reason for the new syntax is only to be backward compatible with the current semantics of A.B (which is to load all of A). Let's put import A.B is not guaranteed to load A to the 2.0 breaking change list? :P

[StefanKarpinski]

If the package opts into it, then changing the behavior of import A.B would be acceptable. If that stands a chance of causing a breaking change to callers, then that's on the package developer.

[KristofferC]

Yeah, that's probably a better version of opting into it than the caller.

[StefanKarpinski]

There are several counter-arguments to the need for this feature in the first place:

1. If you can import B without importing A and B is independent enough that it makes sense to do so, then B should be separate from A anyway.

2. It significantly overlaps with the concept of package namespaces if you just say that import A@B is the syntax for loading package B from namespace A, which has had import A/B proposed as a syntax. How many of these slightly different ways to import a package do we really want? Arguably with import and using we already have too many that are too subtly different.

Again: this does not mean that I reject the problem, just some shapes of solutions. Introducing a package namespace concept is slightly more palatable since it's a fairly clear concept, as opposed to subpackages, which seems a bit contradictory: "a package is a unit of reusable code"; "a subpackage is an even smaller unit of reusable code". From the namespace perspective, if we see import A.B and we know that A is a namespace, then we can load B as a package. That's just a code loading feature. The same sort of thing could be done for "subpackages", but again, I don't think we should introduce two such similar features unless we can figure out one design that solves both problems.

[KristofferC]

I wouldn't get too stuck on the particular names I picked for everything. To me, it doesn't matter if it is called a "subpackage", a package B namespaced by A, a conditional part of package A etc. The question is, what does import A.B actually do (assuming the package opted in to it) and what it can be used for. In this proposal, it loads src/B.jl and precompiles that into its own .ji file using the same Project + Manifest as A would have used. What it can be used for is e.g. conditional loading of dependencies where import A.GPU gives you GPU support but is not loaded when you just du import A.

I don't see how it overlaps with the package namespace proposal at all. That is only about how one can disambiguate the string -> UUID lookup that we have. So instead of only having add A resolve to the UUID with name A in the registry you could have add NameSpace/A which finds the UUID of that some other way. It doesn't do anything with code loading which this proposal is fundamentally about. Basically, the namespace proposal will only touch the code in the package manager and registry, this proposal will only touch code in code loading.

[StefanKarpinski]

The overlap is that they both introduce import A.B (where the . might be a different symbol) for which B is the package and A is something that is not a package but some kind of collection of packages which does not need to be loaded. Focusing on the different mechanics of the two is missing the point that there's a conceptual commonality here and it's kind of questionable to introduce two different mechanisms for accomplishing similar things. If I want import A.B to only load code for B but not A I have two different options: make B a subpackage of A or make B a package in the namespace A. Which should I do?

[ma-laforge]

Sorry. I didn't notice you replied.

Package namespaces. Yes. That sounds great to me. I don't really need the subpackage concept. I understand that might end up causing subtle issues. That was not the intent.

Need:

• What I want is a library of packages that can be bundled in a single .git repository. Package namespaces sound like they could do exactly what I need!
• I want to be able to publish a bunch of packages in a "library" (package namespace) simultaneously. I would like to simplify the process of publishing a "library" of packages for solutions where more than one of these packages often need to be in sync for the "client software" to work properly.
• It would be nice if I only had to register a single package namespace in the Julia registry. After that, only select packages of that namespace could be added to the active environment, depending on its needs.
• It would also be nice if all the packages in that namespace were versioned together (so, I guess the version would belong to the namespace). I think that if developers want individual packages to have different versions in a namespace, well they probably shouldn't be bundling them as a single library.