Pkg.jl: New stdlibs make the resolver sad on earlier Julia versions

[KristofferC]

There are two problems with new stdlibs in Julia and package management when running things with an older Julia version:

1. The manifests from the new Julia versions is unusable if it contains a new stdlib. Pkg verifies that it knows about all packages in a manifest and when it can't find the stdlib it will error. We could perhaps relax this check and print a warning instead and backport that to 1.5. Edit: Actually, I don't th ink there is anything we can do about this. Unless the package with the new stdlib dependency conditionally imports the stdlib, it will error anyway.

2. If a package registers a new version that has a dependency on the new stdlib, adding that package will error on earlier julia versions. If I add TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76" (a new stdlib) to C/Crayons/Deps.toml, and try to add that package, on Julia 1.5, it errors with

(@v1.5) pkg> add Crayons
  Resolving package versions...
ERROR: cannot find name corresponding to UUID fa267f1f-6049-4f14-aa54-33bafae1ed76 in a registry

If I then add the dummy entry

fa267f1f-6049-4f14-aa54-33bafae1ed76 = { name = "TOML", path = "T/TOML"}

to the Registry.toml, it will fail because the package has no known versions:

(@v1.5) pkg> add Crayons
  Resolving package versions...
ERROR: Unsatisfiable requirements detected for package TOML [fa267f1f]:
 TOML [fa267f1f] log:
 ├─TOML [fa267f1f] has no known versions!
 └─found to have no compatible versions left with Crayons [a8cc5b0e]

However, if I restrict the version of Crayons that depends on TOML to 1.6 only then it works fine.

So what I think we need to do is:

• Add dummy entries into Registry.toml for the new stdlibs.
• Ensure that packages that depend on the new stdlibs have Julia 1.6 compat. Artifacts can be excluded because it has worked around it by actually registering Artifacts.jl package that works pre 1.6.

cc @staticfloat, @StefanKarpinski

[StefanKarpinski]

To be clear, the problem here is generating a manifest for a newer version of Julia and then trying to use that same manifest with an earlier version of Julia?

[KristofferC]

That is the first minor point. The second one is not about manifests.

[KristofferC]

To put it into other words, as of right now, any package that adds a dependency on one of the new stdlibs will have all its versions be uninstallable on older julia versions.

[StefanKarpinski]

One way to solve the dummy situation is to register a version of the new stdlib that works on older Julia versions. That's what I'm planning on doing with Downloads, which does actually work on Julia 1.3-1.5. Presumably TOML would be usable on older Julia versions as well? Even if not, we could register a version that's never actually used by any installed version since on 1.6 it uses the stdlib and it's incompatible with older Julias.

[KristofferC]

Yes, that's what I meant with

"""
Add dummy entries into Registry.toml for the new stdlibs.
Ensure that packages that depend on the new stdlibs have Julia 1.6 compat. Artifacts can be excluded because it has worked around it by actually registering Artifacts.jl package that works pre 1.6.
""
Regarding TOML I could maybe make it work on old julia versions but there's already a TOML package registered so it's a bit awkward.

[staticfloat]

For Artifacts, the approach I'm taking is to be as backwards-compatible as reasonably possible, so I have an Artifacts package that I just registered, and all it does is re-export Pkg.Artifacts on 1.3-1.5. It won't install on 1.6+, but that's fine, since the name Artifacts gets overridden due to its nature as an stdlib so the same Project.toml (and even Manifest.toml) is usable on 1.5 and 1.6 simultaneously.

I don't see a registered TOML package in General, where is it?

[KristofferC]

Oh, maybe that got removed when we cleansed all pre-1.0 packages! I'll try make TOML work on older versions and I guess we can just register that one as well then.

[KristofferC]

Fixed TOML.jl for 1.0 and registering it: https://github.com/JuliaRegistries/General/pull/21412.

[KristofferC]

Think this is ok now