Pkg.jl: Proposal of Development.toml file

[bhalonen]

I think a third file "Development.toml" should be added to track local development. It would have identical entries to the Manifest.toml file has when it is developing a repo, without polluting the Manifest file. This will make it easy to keep a description of the dependencies.

Load it with the Manifest, and override whatever is in the Manifest.

Could also have a production-test mode, that ignores the Development.toml.

[bhalonen]

I've spent a lot of time attempting to unroll many local devs. This could also be managed through better git management, but fairly involved.

[fredrikekre]

If I understand correctly, this is what a Manifest.toml commited to the package code already does. It is ignored when the package is used as a package and active when the package in question is the main application, for example when developing.

[bhalonen]

If you have private repos, you need the Manifest file to refer to the urls. The Manifest is also used in ] test. It would be nice to be able to run a check if the tests pass on all the non development package paths. ] production-test, or something like that.

[bhalonen]

I run] add https ... quite a bit.

[fredrikekre]

It would be nice to be able to run a check if the tests pass on all the non development package paths. ] production-test, or something like that.

See https://github.com/JuliaLang/Pkg.jl/issues/286

[bhalonen]

Wouldn't work at all for my case with the private repos, but would take care of it for developing against registries.

[bhalonen]

Here is the steps needed to change a directory from local development back to a repo-url without a registry. We do
] rm package
]add https ... package.jl

[fredrikekre]

Ok, so basically your problem is that we can't free to a url then? Note that the rm is not needed, adding with the url is enough.

[bhalonen]

Little background: I often have 10+ packages being developed locally at a time. When I want to do a release, I need to have the Manifest file pointing at all the urls, so that an instantiate pulls all of the dependencies.

I then need to re dev all the packages. We've written a couple functions called undevelop() and redevelop() that create and unroll a .redevelop.packagename.toml file, but it doesn't work perfectly.

I'm going to see if our functions work, if not, I am going to start making a double branch, one for non local references, one for dev.

git checkout branch-main julia> add https: git add/commit git checkout branch-local julia> dev --local https

[bhalonen]

This would sort of handle it, https://github.com/JuliaLang/Pkg.jl/issues/492, but I still think it is extremely bad for local path information to make it into the Manifest file. It shouldn't be possible for deved packages to end up in a file that is in source control. I would not foresee adding a 'Development.toml' to source control.

[StefanKarpinski]

Shouldn't you then temporarily change Manifest.toml and then git stash save it before testing? Note that having local paths in a manifest is perfectly fine for many use cases. For example if you have an application that uses vendored copies of packages. Then the paths to packages should be relative local paths to where each vendored package lives in the application git repo.

[EricForgy]

Hi 👋

It sounds like maybe this could be handled using different environments. Have you considered that?

[StefanKarpinski]

Right, you could push a dev project file onto the front of LOAD_PATH and it would "occlude" the project's own manifest info.

[bhalonen]

Unsure if merge conflicts would arise if you git stash; julia>] up; git pop.

My main request is that dev wouldn't change my Manifest file.

Hacking the LOAD_PATH instead of using the dev command seems strange.

[StefanKarpinski]

You're not "hacking" the LOAD_PATH—this is what it's for. You're asking for a way to overlay a development environment over your project's environment. That's why the load path is stackable. If you put a development environment with dev'd versions of your packages first in the LOAD_PATH and your project next in the LOAD_PATH then you'll use the dev versions but won't have to commit them.

[bhalonen]

@StefanKarpinski When would you recommend using the development approach recommended here https://docs.julialang.org/en/v1/stdlib/Pkg/index.html versus the LOAD_PATH approach?

Also, LOAD_PATH approach doesn't allow for ]activate package or ]test package which is really nice.

I think I am going to make a package that will apply and remove packages to development in the Manifest.

[StefanKarpinski]

When would you recommend using the development approach recommended here https://docs.julialang.org/en/v1/stdlib/Pkg/index.html versus the LOAD_PATH approach?

I don't think those are at odds with each other, are they?

[bhalonen]

Using dev --local (my daily approach) is a totally different approach than cloning all of my packages in a separate folder and adding it to my LOAD_PATH. Or at least it seems like it to me.

[EricForgy]

I'm just learning this stuff myself, but if I understand Stefan's recommendation here, it goes something like this...

You develop a project as usual adding the dependencies you need (whether in the General registry or not). You end up with a Manifest.toml pointing to the various production dependencies.

Now, if you dev one of those dependencies, it modifies your Manifest.toml to point to your dev folder.

So one thing you can do is to create a development directory holding a development Manifest.toml and push this directory to the top of LOAD_PATH so as you are deving, it will pick up this dev Manifest.toml instead of the production Manifest.toml located in your package directory.

For sure, this should work, but it raises some questions like, where should you keep the development Manfiest.toml? Plus, it seems a little tedious / cludgy.

The more I think about it, the more I like this idea of a Development.toml that is only used when you are deving 🤔

Instead of modifying the Manifest.toml, when you dev DependencyA, it creates a new Development.toml and this Development.toml has precedence over the Manifest.toml.

ALTERNATIVELY...

What if instead of replacing a deved dependency location in Manifest.toml, it creates a new entry? 🤔

Grabbing a random entry from one of my manifests:

[[BinDeps]]
deps = ["Compat", "Libdl", "SHA", "URIParser"]
git-tree-sha1 = "12093ca6cdd0ee547c39b1870e0c9c3f154d9ca9"
uuid = "9e28174c-4ba2-5203-b857-d8d62c4213ee"
version = "0.8.10"

Now let's say we dev BinDeps. The manifest is updated to:

[[BinDeps]]
deps = ["Compat", "Libdl", "SHA", "URIParser"]
path = "C:\\Users\\Eric\\.julia\\dev\\BinDeps"
uuid = "9e28174c-4ba2-5203-b857-d8d62c4213ee"
version = "0.8.10+"

It looks like my git-tree-sha1 is replaced with a path.

What if instead of replacing git-tree-sha1 with path, we simply add a devpath? Then the final entry would look like:

[[BinDeps]]
deps = ["Compat", "Libdl", "SHA", "URIParser"]
git-tree-sha1 = "12093ca6cdd0ee547c39b1870e0c9c3f154d9ca9"
devpath = "C:\\Users\\Eric\\.julia\\dev\\BinDeps"
uuid = "9e28174c-4ba2-5203-b857-d8d62c4213ee"
version = "0.8.10"

Now, if I am not deving, it will use git-tree-sha1, but if I am deving, it will use devpath instead.

After that ascii spew, I think the Development.toml is probably a cleaner solution 🤔

Edit: Plus, you could put Development.toml in your .gitignore.

[fredrikekre]

Why not just use a new environment for development? e.g.

$ tree .
.
├── dev
│   ├── Manifest.toml
│   └── Project.toml
├── Manifest.toml
└── Project.toml

1 directory, 4 files

where dev/Manifest.toml has my dependencies in dev-mode, and Manifest.toml has them in production mode? Now, to develop I do

julia --project=dev

to run code in the dev environment, and

julia --project

to run code in production mode.

I don't understand why you need Development.toml, and I don't understand under which circumstances this file would be used. Wouldn't you have to activate it explicitly like in my example above? In that case, why cant the file be dev/Manifest.toml, why does it have to be Development.toml?

[EricForgy]

I don't understand why you need Development.toml, and I don't understand under which circumstances this file would be used.

I can't speak for @bhalonen , but I didn't know you could do that (I always start Julia by clicking it) 😅

Maybe that is the solution? 😊

[StefanKarpinski]

where should you keep the development Manfiest.toml

A named environment, e.g. ~/.julia/environments/dev, would do nicely. Then modify your LOAD_PATH to be ["@dev", "@", "@#.#", "@stdlib"].

The more I think about it, the more I like this idea of a Development.toml that is only used when you are deving 🤔

Frankly, this issue feels very much like someone having a very particular use case and seeing _a_ solution to that use case involving adding another file—which, if everyone got to do that, there would be a lot of random files. Believe me, there are many potential problems for which adding a file seems like it could be an answer. However, the package manager needs to be designed to solve a lot of different use cases with a few orthogonal components. Therefore the bar for creating _another_ config file is _extremely_ high. You can more easily convince me that we need to do almost anything else. Still, if there's a really clear, coherent mental model for why this is the only good way to do it, I could be convinced. But I'm not really getting the impression that alternative approaches to solving the problem using other features that already exist in the package manager are being considered all that seriously.

After that ascii spew, I think the Development.toml is probably a cleaner solution 🤔

I see.

[EricForgy]

That is pretty slick https://github.com/JuliaLang/Pkg.jl/issues/993#issuecomment-454386702 ☝️😍

Would be nice if that got into the docs 😊 (not volunteering right now and I wouldn't do it justice anyway 😅)

[StefanKarpinski]

Note that you can use something like direnv to arrange for your LOAD_PATH to be set to a specific value in the directory of a specific project. That way the load path wouldn't always have to contain @dev, it could contain it only when you cd into your project directory. You could have a direnv script that looks for a dev subdirectory of a project and only modifies LOAD_PATH if it exists. There are many many possibilities with the features that already exist.

[krrutkow]

Therefore the bar for creating another config file is extremely high. You can more easily convince me that we need to do almost anything else.

It seems that another config file is absolutely required for practical usage, whether you use a separate Project/Manifest file for the dev-ed packages or the proposed Development file. Something like a Development file makes sense to protect the Manifest and would eliminate the redundancy and manual effort involved in using a separate Project/Manifest too.

It looks like Pkg users have found different solutions to the dev-ing packages problem since it is not clearly achievable with Pkg alone. If it is not the goal of Pkg to solve this particular problem, then it would benefit the community just to have the approaches (starting with what has been suggested in this thread) listed with strengths and weaknesses of each.

[EricForgy]

I like the Development.toml idea, but I see @fredrikekre 's dev/Manifest.toml as basically the same thing.

The downside of dev/Manifest.toml instead of Development.toml is that you need to change the LOAD_PATH somehow either at the command line when launching julia or by pushing a directory to LOAD_PATH directly.

Many users, especially Windows users, like myself do not launch Julia from the command line so there should be a solution that doesn't depend on that 🤔

Would there be a way - whether with direnv or whatever - that you can modify the load path when any package is deved? In other words, if nothing is deved, it uses Manifest.toml. If anything is deved, it uses dev/Manifest.toml? 🤔

In any case, I am confident there should be some acceptable solution using the current capabilities.

[fredrikekre]

Many users, especially Windows users, like myself do not launch Julia from the command line so there should be a solution that doesn't depend on that

You don't have to do this from the command line, you can do:

activate dev

in the Pkg REPL, or

using Pkg
Pkg.activate("dev")

from the Julia REPL.

[bhalonen]

Similar request over here. https://github.com/JuliaLang/Pkg.jl/issues/867

[bhalonen]

@fredrikekre The biggest downside to the julia --project=dev approach (which is pretty darn cute) is that you need to add packages twice, once to the proj/Manifest.toml and once to the proj/dev/Manifest.toml. Also the somewhat clunky proj/dev/dev folder structure.

[bhalonen]

@StefanKarpinski I doubt there is a compelling argument that anyone could make besides volume, so now that I've registered my request, I'll close the issue. I doubt you'd make any change like this unless there is a lot of people asking for it. the dev/dev approach is just the best we got.

[fredrikekre]

Also the somewhat clunky proj/dev/dev folder structure.

Why not just proj/dev as in https://github.com/JuliaLang/Pkg.jl/issues/993#issuecomment-454386702?

[bhalonen]

@fredrikekre I use dev --local to develop projects because I prefer to use different environments/not step on my toes. So, if I run julia --project=dev; ]dev --local package, it builds the described dev/dev/package folders.

I guess you could always remember to set JULIA_PKG_DEVDIR=joinpath(pwd(),"dev") and run dev package, but that is much less automagic, and also eliminates the ability to have a package under general development (across many repos). I'm nervous about changing things like environment variables because if you walk away from the code for a few months you might forget the magic set of environment variables that get the project up and running.

Unless there is a magic trick to dev into your own local folder, but I am extremely leery of environment hacking beyond the easily understandable.

We are growing our julia development team, and having a standard issue approach that doesn't rely on brilliance and a good memory to do things right is important. I'll reccomend the dev/dev unless I have time to finish off the package I started last night.

By the way, I see you are a computational mechanics guy, I wrote a package that takes a functional approach to digitial image correlation, correlating a polynomial to a displacement field. https://github.com/bhalonen/DIC.jl