Pkg.jl: support ENV in Manifest, make Manifest more portable

[ExpandingMan]

Right now the "portability" of a Manifest.toml is reliant on the assumption that you will essentially be pulling everything from git. If you have instead done add or develop then your Manifest.toml is dependent on your directory structure (even though develop provides a little bit of a work around by giving the relative path). At this point, if you move everything to a new environment (like docker) you will break Julia's ability to locate everything unless you are very careful about how you set up your directory structure.

My suggestion is to provide users with another option by allowing users to provide environment variables, such as

add $PKG_DIR/SomePackage

It should be possible to have the Manifest.toml store this in the same field that it stores the path, it would just require checking strings for environment variables. Or, if this gives rise to too much complexit elsewhere, perhaps we could add root-dir-env or something like that. Of course, in any case, this would require input paths to be parsed for environment variables.

What do you think? Might there be a better solution I'm not thinking of for making Manifest.toml protable without necessarily relying on git?

[fredrikekre]

I still don't quite understand what the setup is like is it possible to provide a simple MWE? The only time the path should matter is when using develop and in those cases you can use the relative path.

[ExpandingMan]

The fact that dev uses a relative path is indeed helpful, and that's what I'm using as a work-around at the moment, but it still causes the Manifest.toml to depend on the directory structure (in this case the relative directory structure). That still means you have no way of making it truly portable without using git.

As for me my real problem is that the private gitlab I need to clone from is being extremely uncooperative. As you can see here I'm beginning to suspect this is actually an issue with LibGit2.

Regardless, even if I could clone the repos I need to, considering the level of absurd security paranoia surrounding it, it seems rather likely I'll be in this position (of not being able to use git) at some point again in the future. I strongly suspect that will be the case for others as well. It would also be nice to have a temporary solution for people using other types of version control.

I'm not necessarily claiming that what I'm suggesting here is the best solution, but from where I sit it still seems like functionality that we ultimately want.

[KristofferC]

I'm a bit confused. A Manifest containing paths is inherently non-reproducible because it depends on the state of the files in the path.

[ExpandingMan]

If you are trying to reproduce a working package somewhere, such as in a docker container, it would be nice to simply be able to move the Manifest.toml without going through package management stuff again (yes, you'd still have to move the files). If the Manifest.toml could see environment variables, this would be easy: as long as those variables are present in both locations and your paths relative to those are correct it would work. If you are relying on a relative path, the relative path has to be the same in the new build as in the original, there is a lot less flexibility for reproducing it.

[00vareladavid]

What about using relative paths in Pkg + symbolic links to locate the directories?

After copying the manifest + dependencies, the only additional work you would have to do is to point the symbolic links to the right directories.

[ExpandingMan]

That would be my one of my usual solutions. I guess the problem in my particular case is that docker was being extremely uncooperative when it comes to symlinks.

Nevertheless, I feel like the bigger issue here is to what extent you have to emulate any environment other than the one you're currently in. I feel like the most common way of dealing with this sort of thing is environment variables. e.g. usually you'd add to your PATH rather than making symlinks in /usr/bin.

[StefanKarpinski]

I don't really understand what ENV means in the title of this issue. You mean like the operating system ENV dict? You want to save that in the manifest file somewhere?

[ExpandingMan]

Yes, using environment variables to locate packages. Providing the locations of packages relative to environment variables.

Is it really still very unclear what I'm after here, having a Manifest.toml that's a little bit portable without necessarily relying on git? Again that's definitely not an ideal situation at all, it's just one that some of us may be stuck with sometimes, and it seems like a common enough thing that perhaps it should be addressed (though I suppose there's a small chance I'm just completely wrong and nobody else will ever try to do this).

[StefanKarpinski]

We could teach Pkg that if path points to a file then read its contents and load the package from the location in the file. So a symlink without the symlink. Would solve the issue on Windows or Docker.

[StefanKarpinski]

Or have link = "file/containing/path" as an indirect alternative to path. If link is a symlink, it would just work, but if its a file then Pkg would open it and read path from it.

[KristofferC]

We could teach Pkg that if path points to a file then read its contents and load the package from the location in the file.

You would have to teach loading.jl that, Pkg is just a stupid Manifest emitter.

Anyway, either, you have absolute paths and then you can transfer the Manifest to wherever that can access that absolute path.

Or you have a "bundle" for example:

    Project.toml
    Manifest.toml
    dev/PkgA
    dev/PkgB

where you have relative paths to PkgA and PkgB and to relocate that project you need to move the whole Bundle folder.

Is the problem here that you want to use the same files that are on your harddrive but because you are in a container environment the absolute paths have changed?

[ExpandingMan]

Yes, essentially that's the problem, how to use packages in specific locations if paths change.

[StefanKarpinski]

You would have to teach loading.jl that

Yeah, yeah. I'm well aware. I think that Pkg would also need to be somewhat aware of this though.

[KristofferC]

I've read this again and I still don't know what the feature required is and how this can be fixed. Closing, feel free to open a new issue with a bit more concrete description of the issue.