Pkg.jl: should `generate` create packages in ~/.julia/dev by default?

[StefanKarpinski]

Currently generate creates a new directory in the current working directory which I found surprising. Perhaps generate HelloWorld could generate in JULIA_PKG_DEVDIR whereas if you give a relative path like generate ./HelloWorld it would generate in the current directory?

[KristofferC]

I don't think it is surprising. I think few people want all their projects to live in .julia.

[StefanKarpinski]

I think there are three likely classes of people:

1. Those who don't develop packages at all.
2. Those who do and keep all the packages they're developing in the default ~/.julia/dev location.
3. Those who do and want to develop packages elsewhere so they set JULIA_PKG_DEVDIR.

In all of these cases generating a new package in the dev dir (default or JULIA_PKG_DEVDIR if set) is the most useful and sensible thing to do. What is the use case for generating in the current directory?

[StefanKarpinski]

Another way to look at it: if gen, which generates a new package for development, generates a project in the current directory, why does dev which checks out an existing package for development, not check it out in the current directory? I could imagine gen being spelled as dev --gen or dev --new instead, at which point the location inconsistency would be very jarring.

[KristofferC]

In all of these cases generating a new package in the dev dir (default or JULIA_PKG_DEVDIR if set) is the most useful and sensible thing to do. What is the use case for generating in the current directory?

Because if you want to start using your project you have to cd to it.

why does dev which checks out an existing package for development, not check it out in the current directory?

Yes, I have been thinking about too, checking out to the a directory inside the current directory likely makes sense if you dev a url.

There is a difference between generate and dev. dev is for when you want to add a temporary hotfix to a dependency of your project so you need to repoint the dependency to a local path. generate is for "long term" when you are developing the package itself, not temporarily swapping out a dependency for a path.

[KristofferC]

We could probably get rid of JULIA_PKG_DEVDIR completely, have people dev urls to whatever local path they want and then if they want to use the same dev version for another project, they can just dev that local path.

I don't really like JULIA_PKG_DEVDIR because it mixes stuff from different projects in a big soup. The reason I made it like this is that people likely want to use the same hotfixed dependency for multiple projects...

[StefanKarpinski]

So there are then a few kinds of people:

1. Those who want to manage development locations themselves and accordingly want dev and gen to operate in the current directory.
2. Those who want all development copies of packages in a single location, either ~/.julia/dev or JULIA_PKG_DEVDIR.
3. Those who want some mix of this – no idea what to do there.

But in any case, the mix of one command doing things in the current directory and the other one doing it in some standard place seems weird to me.

[KristofferC]

Yeah, we should probably just do 1. and remove 2. It is quite bad now, you dev a package in a random project and because another package has deved it, it already exists in .julia/dev so we are using that package which might have a bunch of debug stuff from the original project that deved it. Soon you have a bunch of projects with dependencies pointing to the same path and things are coupled all over.

Shoving everything into a single directory has the advantage that I can just mv ~/.julia/v0.7 ~/.julia/dev though.

[StefanKarpinski]

Ugh, that is my least favorite behavior. I really like having everything neat and tidy in once place, and I really don't (usually) want more than one development copy of any package.

[quinnj]

fwiw, I'd prefer pkg generate to always put packages in .julia/dev. I agree w/ Stefan that having everything always live in the same place is really nice and makes snooping around much easier. Sure, let people set an environment variable to change it, but I think by default, they should always go in .julia/dev.

[KristofferC]

I agree w/ Stefan that having everything always live in the same place is really nice and makes snooping around much easier.

So why are we doing this at all? You can already have things in one place by not using any projects at all, just keep using .julia/v0.7.

[StefanKarpinski]

We're going to stop supporting that in 1.0; looking in the old Pkg.dir is just a transitional thing.

[KristofferC]

Yeah but let's not then. It is clearly "really nice" and makes "snooping around much easier". All packages in one place, ready to be edited at will.

[StefanKarpinski]

So you're totally cool with the inexplicable and pointless name v0.7 for the dev directory?

[KristofferC]

That's why I said "has the advantage that I can just mv ~/.julia/v0.7 ~/.julia/dev though."

[tpapp]

With Pkg2, I kept my "own" general packages indented for public consumption outside ~/.julia/, and I can see how Pkg3 allows me to move everything into one place, which is a nice thing.

But I also keep project-specific code that I just packaged up for convenience (using Revise.jl, unit testing, etc) in directories which are all over the place, inside project-specific directories which also have notes, papers in progress, and occasionally code in other languages. I can live with the workflow of generating them inside ~/.julia/dev/ and mving them manually since I don't do this that often, but doing it in a single step would be nice.

(sorry if I missed something about Pkg3 which makes my comment/workflow nonsensical)

[tshort]

Isn't generate going to be used more for plain projects (those that aren't packages) than for packages? These are projects where I'll add some packages and do some analysis. (I may be missing something with terminology here.) If so, using the local directory makes the most sense.

If I have a plain project where I want to do a package hotfix like Kristoffer describes in the docs, I think I'd want those changes locally, too.

For real package development, I would probably put them somewhere common. I've tended to be gun-shy about putting stuff in .julia because I've often had to nuke that (hopefully, no more nuking with Pkg3).

[KristofferC]

For real package development, I would probably put them somewhere common.

And it is impossible to know, therefore defaulting to pwd (which is what every type of command line command ever does) is the only choice that makes sense.

npm init generates in your pwd.
cargo new generates in your pwd.

Is there any package manager that generates the files into a hidden directory and totally ignores the pwd? If you want to collect your projects to one folder, just be there when you generate it.

[StefanKarpinski]

We could support an optional devdir = "dev" entry in Project.toml to tell the Pkg3 that it should checkout development packages in the dev directory when working on that project.

I would be fine with defaulting to the current working directory, but it seems like having a way to opt into particular locations is something that many people want. It seems like you're leaning towards making dev by default checkout packages in the current directory as well, @KristofferC?

[KristofferC]

The problem I have right now is that it is way too easy (convenient?) to accidentally create couplings between different projects. One dev JSON in one project and one dev JSON in another project and these projects are now linked by a shared mutable dependency. Someone who develops a lot of packages would soon end up in a situation which is similar to the Pkg2 one with all packages coupled to each other and everything mutable. Such a situation should only occur explicitly from someone creating a Project and devving a bunch of packages into that project.

but it seems like having a way to opt into particular locations is something that many people want

How about in .julia/dev if you are not in your own custom project (so you are using the fall back one in `.julia/environments) but in the project path if you are in a project?

[StefanKarpinski]

One dev JSON in one project and one dev JSON in another project and these projects are now linked by a shared mutable dependency.

Is this really so bad? If I'm using the development version of JSON in two different projects, I would generally want to be using the _same_ development version of it. Either (a) I'm a JSON developer and I've got local changes to it that I'll want to test and use in all the projects where I've opted into it or (b) I'm not a JSON developer and I won't have any local changes to it so it's find for all the projects that are using dev JSON to share the same one. I'm having trouble imagining the scenario where this is a problem. I do think we should support having multiple separate dev sets but I don't think that should be the norm.

[KristofferC]

If I'm using the development version of JSON in two different projects

This should imo be done by adding the local repo of JSON, which would track the master branch by default, and then you update like normally when you want to update to a new commit. Not by using dev.

dev is for temporarily repointing a dependency when you need to try something out and want changes to the code in the dependency immediately take effect (pretty much only needed when interactively working on it). It should perhaps be renamed. Perhaps override.

[StefanKarpinski]

I think we should maybe step back and think about what we want the high-level meanings of operations like dev JSON to be. Currently, it has nothing to do with the current environment – it just checks out a git clone of JSON _somewhere_ and has no effect on the current environment. This is a bit strange since presumably we'd also want to add JSON if it's not already, no? Checking it out in the current directory doesn't really help that at all either – it just creates a random subdirectory inside of one project where it puts a clone of another project.

[KristofferC]

It does add it as well?

pkg> dev JSON
   Cloning package from https://github.com/JuliaIO/JSON.jl.git
 Resolving package versions...
  Updating `Project.toml`
 [682c06a0] + JSON v0.17.1+ [`~/.julia/dev/JSON`]

It didn't for you?

[StefanKarpinski]

Ah, I didn't realize that it did that as well.

[KristofferC]

How about a --local, --global option to develop? local puts it under your project, global puts it in some default folder overridable by the env variable.

[StefanKarpinski]

Yes, I like that. Maybe --local versus --shared? I've found that global confuses people for some reason.

[tpapp]

Revisiting this issue, I wonder if someone could clarify the current recommended workflow for

1. starting a new package, or continuing to develop an existing one, with a git repo in a directory
2. supporting multiple versions (currently v0.6 and v0.7) from a single directory / git repo.

Eg if I

julia> cd(expanduser("~/.julia/dev"))
(v0.7) pkg> generate MyPackage

then I will have a package skeleton in ~/.julia/dev/MyPackage, but

julia> using MyPackage

will fail because it is not found in ~/.julia/environments/v0.7/Project.toml. I could not figure out how to add it either.

Should I start "local" packages in another directory? (related discussion on Discourse)

[KristofferC]

.julia/dev has no special meaning or logic when it comes to loading packages. It is just the default location where packages that are devved from Pkg are put. A new package can reside wherever and the way you add it to the project is by dev localpath where localpath is the path to the package.

[tpapp]

So should I just

1. keep/generate my own packages in a directory outside ~/.julia/,
2. put that in LOAD_PATH for v0.6 and v0.7,
3. in v0.7, pkg> dev /wherever/the/package/is?

and this will just use the package from that path directly?

Eg

julia> LOCAL_PACKAGES # comes from my startup file
"/home/tamas/src/julia-local-packages/"

julia> Pkg.API.generate(joinpath(LOCAL_PACKAGES, "MyPkg"))
Generating project MyPkg:
    /home/tamas/src/julia-local-packages/MyPkg/Project.toml
    /home/tamas/src/julia-local-packages/MyPkg/src/MyPkg.jl

julia> Pkg.API.develop(joinpath(LOCAL_PACKAGES, "MyPkg"))
 Resolving package versions...
  Updating `~/.julia/environments/v0.7/Project.toml`
  [3defccca] + MyPkg v0.1.0 [`~/src/julia-local-packages/MyPkg`]
  Updating `~/.julia/environments/v0.7/Manifest.toml`
  [3defccca] + MyPkg v0.1.0 [`~/src/julia-local-packages/MyPkg`]

[ssfrr]

I'm about to do some 0.7 development of a few related packages, so I

1. created a new ~/audiodev directory
2. ran pkg> init to create a new environment I'll use to isolate this development
3. ran pkg> dev RingBuffers PortAudio LibSndFile. I was surprised that it cloned the packages to a shared ~/.julia/dev directory

What if generate and dev both created/cloned the packages in a dev directory within the current environment? It seems like if I went to the trouble of creating a new environment I want some isolation, so it seems weird for different environments to share the same package repo state. It's also nice because then I can just delete the whole environment directory if I don't want it anymore, and it will also remove any packages I was hacking on as part of that environment.

I'd also support dev cloning into pwd(), but it seems like Pkg is moving away from depending as much on the current directory.

[KristofferC]

Yeah indeed, the default dev dir should probably be in the current project with a shared option to put it where it currently goes.

But putting things in .julia/environments/v0.7/dev seems a bit to deep. Perhaps if one uses the named environment we out it where it is now and otherwise in the project?

[KristofferC]

I think this should put things by default into the HOME_PROJECT. Opinions @StefanKarpinski?

[StefanKarpinski]

If we go with devving into the “current project” then the right place would seem to be Base.active_project() since that is what all package actions operate on. Otherwise it would be entirely possible to dev a package and not be able to load it immediately.

[KristofferC]

Yeah, that's sort of what I meant but didn't write :P

[StefanKarpinski]

I also think that ~/.julia/environments/v0.7/dev does seem to be the natural place to put shared dev packages in this approach but agree that it's kind of absurdly deep. Maybe we can just make that a symlink to ~/.julia/dev by default or something?

[KristofferC]

I think the status quo is ok for generate, possible that we should be more explicit about using PkgTempaltes in the docs.

[StefanKarpinski]

Maybe we should remove it and just tell people to use PkgTemplates instead?