Pkg.jl: Allow activating custom project targets

[mortenpi]

I feel that Pkg should allow activating one of the [targets] as an environment. It has probably come up before but I didn't find an issue for this. For the user there probably should be:

• Relevant Pkg / REPL API: activate --target=foo <path> ?
• A way to activate from the command line: julia --project=<project> --project-target=foo or maybe something like julia --project=<project>:foo

Although.. since --project can be a path, then having some sort of a separator would be tricky?

[StefanKarpinski]

100% agree, although the design and implementation details still need to be worked out.

[KristofferC]

Yes, this is a good idea, however, you would need to do a resolve to activate it.

[StefanKarpinski]

What I'd like to do is:

1. Rewrite the resolver as a SAT solver that can resolve target dependencies correctly at the same time.
2. Record extra dependencies in the manifest as well as main dependencies, but don't necessarily install those unless requested.
3. When activating a target, you already know from the main manifest which versions to use for the extras, you just have to tell it not to ignore those marked as extra.

[mortenpi]

So the extra dependencies will also limit the versions you can install in the main environment? And the bounds on the extra deps of a package's dependency will also be checked when installing a package? (Not saying this would be bad, just curious.)

[StefanKarpinski]

Yes, that's the general idea.

[00vareladavid]

I just read an article about SAT solvers and immediately thought of Pkg. I'm curious how serious this plan is? IIRC performance was a concern, how do you think it would compare to the current resolver? Do you have an existing implementation in mind?

[StefanKarpinski]

I'm planning on implementing this paper in Julia. They hacked up MiniSAT to evaluate their implementation but the patch isn't public or, from the sound of it, super clean. I suspect that implementing a SAT with preferences solver in Julia won't be all that hard compared to doing it in C, for example.

[00vareladavid]

Do you know of any good test sets? I have a basic version of the minisat algorithm running in julia. Its passing for hand coded tests, but its better if I test against tests I didn't write myself. I found tests for the SAT competition, but they are quite large and AFAICT only test if the model is SAT/UNSAT (i.e. they dont provide concrete solutions). If nothing else, I will have to map some problem (e.g. Sodoku) to provide small tests suitable for debugging/introspection.

[oxinabox]

related:
https://github.com/JuliaLang/Pkg.jl/issues/744

[simonbyrne]

a simpler solution would be to generate additional manifest files, say Manifest.targetname.toml?

[o314]

a simpler solution would be to generate additional manifest files, say Manifest.targetname.toml?

Since v1.2, a package can have multiple Project.toml too.
One for src, one for test optionally.
May be one for docs ...

That has to be taken into consideration regarding the number of manifest files.

--

Other approach : if we allow multiple projects per package, is target still useful ?

[o314]

FYI, the following package manager (at least) are driven by sat

• composer, package manager for php, license mit

  • https://github.com/composer/composer
  • dependency resolution with a sat solver

• zypp, package manager for opensuse/suse, license gpl

  • https://github.com/openSUSE/libzypp
  • https://doc.opensuse.org/projects/libzypp/HEAD/
  • using sat for solving package dependencies

• libsolv, package manager for opensuse/suse, license bsb

  • https://github.com/openSUSE/libsolv

• deps, package manager for go lang, license bsd

  • https://github.com/golang/dep/
  • https://golang.github.io/dep/docs/introduction.html
  • https://github.com/golang/dep/blob/master/docs/the-solver.md

[pdeffebach]

Has there been progress on this issue? Would be nice to have.

[oschulz]

Just wanted to vote for this (I understand it's not easy, though, from what Stefan writes). Found myself wishing for it today (https://discourse.julialang.org/t/activating-test-dependencies/48121/4) when trying to debug a package and it's tests.

[timholy]

I don't understand why it requires a whole new resolver algorithm, and I worry that this has distracted the focus from an easier solution. It seems that what you need to do is create a temporary environment that incorporates the [deps] and [extras] of the "parent" environment and then run resolve & instantiate on the temporary environment.

Now that we're about to have a TOML cache this seems easier than ever?

[oschulz]

When I asked for this on discourse, I had assumed the mechanism was already there - after all, Pkg.test does activate the test dependencies. I guess that uses a temporary environment or so?

[timholy]

I don't know how it works, but for this case you need to have a temporary environment that persists at the REPL.

[fredrikekre]

https://github.com/JuliaLang/Pkg.jl/issues/1233 is the plan, we can probably close this issue in favor of that.

[o314]

@00vareladavid

I have a basic version of the minisat algorithm running in julia. Its passing for hand coded tests, but its better if I test against tests I didn't write myself. I found tests for the SAT competition, but they are quite large and AFAICT only test if the model is SAT/UNSAT (i.e. they dont provide concrete solutions). If nothing else, I will have to map some problem (e.g. Sodoku) to provide small tests suitable for debugging/introspection.

Have you already published it ? I will be glad to have a look at it ...